[jira] [Commented] (HDFS-8668) Erasure Coding: revisit buffer used for encoding and decoding.
[
https://issues.apache.org/jira/browse/HDFS-8668?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048357#comment-15048357
]
Hadoop QA commented on HDFS-8668:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
21s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
45s {color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
45s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m 1s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
48s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 6m
44s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 57s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 50s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 1m 39s
{color} | {color:red} hadoop-common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
42s {color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 42s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
37s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 37s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
11s {color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 33s
{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m 37s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
47s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 32s
{color} | {color:red} hadoop-hdfs-client in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m 34s
{color} | {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 50s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 48s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s {color}
| {color:red} hadoop-hdfs-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 34s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} |
[jira] [Created] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
Fei Hui created HDFS-9530:
-
Summary: huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
Key: HDFS-9530
URL: https://issues.apache.org/jira/browse/HDFS-9530
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Fei Hui
i run a hive job, and errors are as follow
===
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException:
org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
processing row {"k":"1","v":1}
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while processing row {"k":"1","v":1}
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
at org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
... 8 more
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
could only be replicated to 0 nodes instead of minReplication (=1). There are
3 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
at
org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at
org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
at
org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
at
org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
at
org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
... 9 more
Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
/test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
could only be replicated to 0 nodes instead of minReplication (=1). There are
3 datanode(s) running and no node(s) are excluded in this operation.
at
org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
at
org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
at
org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
at
[jira] [Commented] (HDFS-9458) TestBackupNode always binds to port 50070, which can cause bind failures.
[
https://issues.apache.org/jira/browse/HDFS-9458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048295#comment-15048295
]
Hadoop QA commented on HDFS-9458:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 2s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 11s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
26s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 29s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
3s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 3s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
37s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 34s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 28s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 17s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 1s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 24s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 198m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.TestDFSStripedOutputStreamWithFailure200 |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.shortcircuit.TestShortCircuitCache |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes |
| | hadoop.hdfs.server.datanode.TestDirectoryScanner |
| | hadoop.hdfs.server.namenode.TestAddOverReplicatedStripedBlocks |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.server.blockmanagement.TestPendingInvalidateBlock |
| | hadoop.hdfs.TestEncryptionZones |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.qjournal.TestSecureNNWithQJM |
| |
[jira] [Commented] (HDFS-7866) Erasure coding: NameNode manages EC schemas
[
https://issues.apache.org/jira/browse/HDFS-7866?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048347#comment-15048347
]
Rui Li commented on HDFS-7866:
--
I tried to add another system policy (with different schema) and hit the
following problem.
In {{FSDirErasureCodingOp::getErasureCodingPolicyForPath}}, if the inode is a
file, we retrieve the policy by a numeric ID:
{{INodeFile::getErasureCodingPolicyID}}.
For directories however, the policy is retrieved by name:
{code}
final XAttrFeature xaf = inode.getXAttrFeature();
if (xaf != null) {
XAttr xattr = xaf.getXAttr(XATTR_ERASURECODING_POLICY);
if (xattr != null) {
ByteArrayInputStream bIn = new
ByteArrayInputStream(xattr.getValue());
DataInputStream dIn = new DataInputStream(bIn);
String ecPolicyName = WritableUtils.readString(dIn);
return fsd.getFSNamesystem().getErasureCodingPolicyManager().
getPolicy(ecPolicyName);
}
}
{code}
Therefore the newly added policy works for dirs but not files. I think we need
a unified way to store/get a policy. Any thoughts?
> Erasure coding: NameNode manages EC schemas
> ---
>
> Key: HDFS-7866
> URL: https://issues.apache.org/jira/browse/HDFS-7866
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Kai Zheng
>Assignee: Rui Li
> Attachments: HDFS-7866-v1.patch, HDFS-7866-v2.patch,
> HDFS-7866-v3.patch
>
>
> This is to extend NameNode to load, list and sync predefine EC schemas in
> authorized and controlled approach. The provided facilities will be used to
> implement DFSAdmin commands so admin can list available EC schemas, then
> could choose some of them for target EC zones.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9472) concat() API does not resolve the .reserved path
[
https://issues.apache.org/jira/browse/HDFS-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-9472:
---
Attachment: HDFS-9472-02.patch
> concat() API does not resolve the .reserved path
>
>
> Key: HDFS-9472
> URL: https://issues.apache.org/jira/browse/HDFS-9472
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9472-00.patch, HDFS-9472-01.patch,
> HDFS-9472-02.patch
>
>
> dfs#concat() API doesn't resolve the {{/.reserved/raw}} path. For example,
> if the input paths of the form {{/.reserved/raw/ezone/a}} then this API
> doesn't work properly. IMHO will discuss here to support this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9472) concat() API does not resolve the .reserved path
[
https://issues.apache.org/jira/browse/HDFS-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048274#comment-15048274
]
Rakesh R commented on HDFS-9472:
Thanks [~umamaheswararao]. Agreed, attached another patch addressing the
comments.
> concat() API does not resolve the .reserved path
>
>
> Key: HDFS-9472
> URL: https://issues.apache.org/jira/browse/HDFS-9472
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9472-00.patch, HDFS-9472-01.patch,
> HDFS-9472-02.patch
>
>
> dfs#concat() API doesn't resolve the {{/.reserved/raw}} path. For example,
> if the input paths of the form {{/.reserved/raw/ezone/a}} then this API
> doesn't work properly. IMHO will discuss here to support this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9505) HDFS Architecture documentation needs to be refreshed.
[ https://issues.apache.org/jira/browse/HDFS-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048320#comment-15048320 ] Masatake Iwasaki commented on HDFS-9505: I would like to start working on this. If someone is already on this, ping me please. > HDFS Architecture documentation needs to be refreshed. > -- > > Key: HDFS-9505 > URL: https://issues.apache.org/jira/browse/HDFS-9505 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Chris Nauroth >Priority: Minor > > The HDFS Architecture document is out of date with respect to the current > design of the system. > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > There are multiple false statements and omissions of recent features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-9505) HDFS Architecture documentation needs to be refreshed.
[ https://issues.apache.org/jira/browse/HDFS-9505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Masatake Iwasaki reassigned HDFS-9505: -- Assignee: Masatake Iwasaki > HDFS Architecture documentation needs to be refreshed. > -- > > Key: HDFS-9505 > URL: https://issues.apache.org/jira/browse/HDFS-9505 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Chris Nauroth >Assignee: Masatake Iwasaki >Priority: Minor > > The HDFS Architecture document is out of date with respect to the current > design of the system. > http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs/HdfsDesign.html > There are multiple false statements and omissions of recent features. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
[
https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048384#comment-15048384
]
Fei Hui commented on HDFS-9530:
---
i want to add logs to monitor DataNode DFS Used, Non DFS Used,and so on.
how can i do that ?which files i need to modify?
> huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
> -
>
> Key: HDFS-9530
> URL: https://issues.apache.org/jira/browse/HDFS-9530
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>
> i run a hive job, and errors are as follow
> ===
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
> ... 9 more
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
>
[jira] [Created] (HDFS-9531) Can't use windows network drive
tian created HDFS-9531: -- Summary: Can't use windows network drive Key: HDFS-9531 URL: https://issues.apache.org/jira/browse/HDFS-9531 Project: Hadoop HDFS Issue Type: Bug Components: fs Affects Versions: 2.6.0 Reporter: tian Priority: Minor When we create a Path like "\\SIMPLESHARE\MyHome$", the double slash will be normalised to single slash. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9523: - Summary: libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail (was: libhdfs++: running under docker causes CI unit tests to fail) > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9472) concat() API does not resolve the .reserved path
[
https://issues.apache.org/jira/browse/HDFS-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048605#comment-15048605
]
Hadoop QA commented on HDFS-9472:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
41s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 13s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 51s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
48s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 38s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 53s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
16s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 50s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 67m 3s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 54s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 181m 51s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.TestNameNodeMetadataConsistency |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestLargeBlock |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12776506/HDFS-9472-02.patch |
| JIRA Issue | HDFS-9472 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs checkstyle |
| uname | Linux 8bcfeef7e6cc 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014
[jira] [Commented] (HDFS-9448) Enable valgrind for libhdfspp unit tests
[ https://issues.apache.org/jira/browse/HDFS-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048535#comment-15048535 ] Hadoop QA commented on HDFS-9448: - (!) A patch to the testing environment has been detected. Re-executing against the patched versions to perform further tests. The console is at https://builds.apache.org/job/PreCommit-HDFS-Build/13813/console in case of problems. > Enable valgrind for libhdfspp unit tests > > > Key: HDFS-9448 > URL: https://issues.apache.org/jira/browse/HDFS-9448 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9448.HDFS-8707.000.patch, > HDFS-9448.HDFS-8707.001.patch, HDFS-9448.HDFS-8707.002.patch, > HDFS-9448.HDFS-8707.003.patch, HDFS-9448.HDFS-8707.004.patch, > HDFS-9448.HDFS-8707.005.patch > > > We should have a target that runs the unit tests under valgrind if it is > available on the target machine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
[
https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048653#comment-15048653
]
Brahma Reddy Battula commented on HDFS-9530:
[~ferhui] thanks for reporting this. Can you please check {{reservedSpace}} and
{{reservedSpaceForReplicas}} values in JMX ( you can check through
http::/jmx).
> huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
> -
>
> Key: HDFS-9530
> URL: https://issues.apache.org/jira/browse/HDFS-9530
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>
> i run a hive job, and errors are as follow
> ===
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
> ... 9 more
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
>
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048654#comment-15048654
]
Allen Wittenauer commented on HDFS-8578:
... except that message only shows up when running under Docker.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048563#comment-15048563 ] Bob Hansen commented on HDFS-9523: -- Removing the line from /etc/hosts that reads :::1 localhost ip6-localhost ip6-loopback forcing the test to connect to the v4 localhost causes the test to pass > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9448) Enable valgrind for libhdfspp unit tests
[
https://issues.apache.org/jira/browse/HDFS-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048617#comment-15048617
]
Hadoop QA commented on HDFS-9448:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
29s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 39s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 40s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 11s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 12s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 5s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 5s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 4m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 4m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 4m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 18s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
9s {color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m 1s
{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 13s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 14s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 38s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 31s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
31s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 64m 15s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12776528/HDFS-9448.HDFS-8707.005.patch
|
| JIRA Issue | HDFS-9448 |
| Optional Tests | asflicense shellcheck compile javac javadoc mvninstall
mvnsite unit xml cc |
| uname | Linux 843b26bdd356 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality |
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048644#comment-15048644
]
Vinayakumar B commented on HDFS-8578:
-
[~aw], yetus is not actually running in docker.
{noformat}JAVA_HOME: /home/jenkins/tools/java/jdk1.7.0_79 does not exist.
Dockermode: attempting to switch to another.
Running in Jenkins mode
Processing: HDFS-8578{noformat}
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048664#comment-15048664
]
Allen Wittenauer commented on HDFS-8578:
You're misinterpreting the output entirely.
a. The first message means the value of JAVA_HOME that was inherited from
pre-Docker doesn't work in Docker. Since Yetus is running under Docker and
wasn't told to use anything different, it's going to find a new one to use
instead.
b. Oh, --jenkins was passed, so let's turn on all the specific bits that are
enabled when running under Jenkins *regardless* of whether this is Docker or
not.
You'll note:
{code}
apache-yetus-201a378/shelldocs/
apache-yetus-201a378/shelldocs/shelldocs.py
apache-yetus-201a378/yetus-project/
apache-yetus-201a378/yetus-project/pom.xml
Running in Jenkins mode
Processing: HDFS-8578
HDFS-8578 patch is being downloaded at Tue Dec 8 10:29:37 UTC 2015 from
https://issues.apache.org/jira/secure/attachment/12776283/HDFS-8578-13.patch
{code}
... that Yetus tell you it is running in Jenkins mode immediately after
untarring too. Jenkins and Docker are not exclusive modes to each other.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-8578:
Attachment: HDFS-8578-14.patch
HDFS-8578-branch-2.7-002.patch
Fixed findbugs and checkstyles
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048659#comment-15048659
]
Vinayakumar B commented on HDFS-8578:
-
I mean. Its falling back to jenkins mode due to Java missing in image.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9448) Enable valgrind for libhdfspp unit tests
[ https://issues.apache.org/jira/browse/HDFS-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9448: - Attachment: HDFS-9448.HDFS-8707.005.patch Path 005: removed gdb from Docker image; it apparently caused yetus to fail to build the container. > Enable valgrind for libhdfspp unit tests > > > Key: HDFS-9448 > URL: https://issues.apache.org/jira/browse/HDFS-9448 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9448.HDFS-8707.000.patch, > HDFS-9448.HDFS-8707.001.patch, HDFS-9448.HDFS-8707.002.patch, > HDFS-9448.HDFS-8707.003.patch, HDFS-9448.HDFS-8707.004.patch, > HDFS-9448.HDFS-8707.005.patch > > > We should have a target that runs the unit tests under valgrind if it is > available on the target machine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8836) Skip newline on empty files with getMerge -nl
[
https://issues.apache.org/jira/browse/HDFS-8836?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048702#comment-15048702
]
Kanaka Kumar Avvaru commented on HDFS-8836:
---
Thanks for taking a look at patch [~ajisakaa].
I think private and accessors are not required for {{skipEmptyFileDelimiter}}
as this variable required to be accessed only in this or a subclass. Also other
variables like {{delimiter}} doesn't have accessors.
> Skip newline on empty files with getMerge -nl
> -
>
> Key: HDFS-8836
> URL: https://issues.apache.org/jira/browse/HDFS-8836
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs-client
>Affects Versions: 2.6.0, 2.7.1
>Reporter: Jan Filipiak
>Assignee: Kanaka Kumar Avvaru
>Priority: Trivial
> Attachments: HDFS-8836-01.patch, HDFS-8836-02.patch,
> HDFS-8836-03.patch, HDFS-8836-04.patch, HDFS-8836-05.patch, HDFS-8836-06.patch
>
>
> Hello everyone,
> I recently was in the need of using the new line option -nl with getMerge
> because the files I needed to merge simply didn't had one. I was merging all
> the files from one directory and unfortunately this directory also included
> empty files, which effectively led to multiple newlines append after some
> files. I needed to remove them manually afterwards.
> In this situation it is maybe good to have another argument that allows
> skipping empty files.
> Thing one could try to implement this feature:
> The call for IOUtils.copyBytes(in, out, getConf(), false); doesn't
> return the number of bytes copied which would be convenient as one could
> skip append the new line when 0 bytes where copied or one would check the
> file size before.
> I posted this Idea on the mailing list
> http://mail-archives.apache.org/mod_mbox/hadoop-user/201507.mbox/%3C55B25140.3060005%40trivago.com%3E
> but I didn't really get many responses, so I thought I my try this way.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9445) Deadlock in datanode
[
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048737#comment-15048737
]
Kihwal Lee commented on HDFS-9445:
--
+1. I will commit it this afternoon, if no one objects until then.
> Deadlock in datanode
>
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Walter Su
>Priority: Blocker
> Attachments: HDFS-9445.00.patch, HDFS-9445.01.patch,
> HDFS-9445.02.patch
>
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block
> BP-x:blk_123_456]":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> "Thread-565":
> waiting for ownable synchronizer 0xd55613c8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8894) Set SO_KEEPALIVE on DN server sockets
[ https://issues.apache.org/jira/browse/HDFS-8894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048708#comment-15048708 ] Kanaka Kumar Avvaru commented on HDFS-8894: --- As commented earlier, the check style issues and no test added errors can be ignored. Test failures and asflicense seems not related to this patch changes > Set SO_KEEPALIVE on DN server sockets > - > > Key: HDFS-8894 > URL: https://issues.apache.org/jira/browse/HDFS-8894 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 2.7.1 >Reporter: Nathan Roberts >Assignee: Kanaka Kumar Avvaru > Attachments: HDFS-8894-01.patch, HDFS-8894-01.patch, > HDFS-8894-02.patch, HDFS-8894-03.patch, HDFS-8894-04.patch > > > SO_KEEPALIVE is not set on things like datastreamer sockets which can cause > lingering ESTABLISHED sockets when there is a network glitch. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9523: - Status: Patch Available (was: Open) > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048850#comment-15048850
]
Hadoop QA commented on HDFS-8578:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
53s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 50s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 46s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
17s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 16s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 51s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 15s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 2s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 61m 47s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 55m 3s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 20s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 158m 12s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.datanode.TestBlockScanner |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestBlockReaderLocal |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| JDK v1.7.0_91 Failed junit tests |
hadoop.hdfs.server.datanode.TestDataNodeRollingUpgrade |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.server.namenode.TestProcessCorruptBlocks |
| | hadoop.hdfs.server.namenode.TestCacheDirectives |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
|
[jira] [Commented] (HDFS-9448) Enable valgrind for libhdfspp unit tests
[ https://issues.apache.org/jira/browse/HDFS-9448?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048867#comment-15048867 ] Bob Hansen commented on HDFS-9448: -- That patch looks good. All of the valgrind tests run under CI (except for the static shim test, which is addressed in HDFS-9523). [~aw]: Does this patch pass muster from your end? > Enable valgrind for libhdfspp unit tests > > > Key: HDFS-9448 > URL: https://issues.apache.org/jira/browse/HDFS-9448 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9448.HDFS-8707.000.patch, > HDFS-9448.HDFS-8707.001.patch, HDFS-9448.HDFS-8707.002.patch, > HDFS-9448.HDFS-8707.003.patch, HDFS-9448.HDFS-8707.004.patch, > HDFS-9448.HDFS-8707.005.patch > > > We should have a target that runs the unit tests under valgrind if it is > available on the target machine. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[
https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048912#comment-15048912
]
Hadoop QA commented on HDFS-9523:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 12m
54s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 17s
{color} | {color:green} HDFS-8707 passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 6s
{color} | {color:green} HDFS-8707 passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 23s
{color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
17s {color} | {color:green} HDFS-8707 passed {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 5m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 22s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 51s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 5m 39s {color}
| {color:red} hadoop-hdfs-native-client in the patch failed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
34s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 50m 20s {color}
| {color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12776585/HDFS-9523.HDFS-8707.000.patch
|
| JIRA Issue | HDFS-9523 |
| Optional Tests | asflicense compile cc mvnsite javac unit |
| uname | Linux 9783a8237236 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | HDFS-8707 / 7ad9b77 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/13815/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.8.0_66.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/13815/artifact/patchprocess/patch-unit-hadoop-hdfs-project_hadoop-hdfs-native-client-jdk1.7.0_91.txt
|
| JDK v1.7.0_91 Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/13815/testReport/ |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-native-client U:
hadoop-hdfs-project/hadoop-hdfs-native-client |
| Max memory used | 76MB |
| Powered by | Apache Yetushttp://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13815/console |
This message was automatically generated.
> libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
>
[jira] [Commented] (HDFS-9281) Change TestDeleteBlockPool to not explicitly use File to check block pool existence.
[
https://issues.apache.org/jira/browse/HDFS-9281?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15048937#comment-15048937
]
Colin Patrick McCabe commented on HDFS-9281:
Thanks, [~eddyxu].
{{checkBlockPool}} feels like an awkward name. It makes me think this function
is doing something to validate the contents of the block pool-- when in fact,
it's just checking for existence. And the fact that it can also check for
non-existence if {{false}} is passed makes it even more confusing.
How about splitting this into a {{verifyBlockPoolExists}} function for
verifying that the block pool exists, and a {{verifyBlockPoolMissing}} function
for verifying that the block pool is missing?
> Change TestDeleteBlockPool to not explicitly use File to check block pool
> existence.
>
>
> Key: HDFS-9281
> URL: https://issues.apache.org/jira/browse/HDFS-9281
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: test
>Affects Versions: 2.7.1
>Reporter: Lei (Eddy) Xu
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Attachments: HDFS-9281.00.patch, HDFS-9281.02.patch,
> HDFS-9281.combo.00.patch
>
>
> {{TestDeleteBlockPool}} checks the existence of a block pool by checking the
> directories in the file-based block pool exists. However, it does not apply
> to non file based fsdataset.
> We can fix it by abstracting the checking logic behind {{FsDatasetTestUtils}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen updated HDFS-9523: - Attachment: HDFS-9523.HDFS-8707.000.patch Patch: have the RpcConnection try all of the endpoints returned by resolve before reporting a failure. > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bob Hansen reassigned HDFS-9523: Assignee: Bob Hansen > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9529) Extend Erasure Code to support POWER Chip acceleration
[
https://issues.apache.org/jira/browse/HDFS-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049101#comment-15049101
]
Zhe Zhang commented on HDFS-9529:
-
I agree. This change most likely will happen under {{hadoop-common-project}}
> Extend Erasure Code to support POWER Chip acceleration
> --
>
> Key: HDFS-9529
> URL: https://issues.apache.org/jira/browse/HDFS-9529
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: wqijun
>Assignee: wqijun
> Fix For: 3.0.0
>
>
> Erasure Code is a very important feature in new HDFS version. This JIRA will
> focus on how to extend EC to support multiple types of EC acceleration by C
> library and other hardware method, like GPU or FPGA. Compared with
> Hadoop-11887, this JIRA will more focus on how to leverage POWER Chip
> capability to accelerate the EC calculating.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7916) 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop
[
https://issues.apache.org/jira/browse/HDFS-7916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049200#comment-15049200
]
Yongjun Zhang commented on HDFS-7916:
-
HI [~rushabh.shah],
Thanks for your reply and sorry I missed this update until now that I'm looking
at a related issue. I created HDFS-9532 when I look at the related code here.
One thing I'd like to check with you and [~vinayrpet]: the HDFS-7916 fix tries
to handle StandbyException correctly as reported, but the fix catches
RemoteException and did not check whether the exception wrapped by the
RemoteException is StandbyException or not. Is it intended to handle all
exceptions wrapped by RemoteException the same way as StandbyException? Is
there any case that we don't want to do the same? It seems worth some
understanding here. Would you guys please comment?
Thanks.
> 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for
> infinite loop
> --
>
> Key: HDFS-7916
> URL: https://issues.apache.org/jira/browse/HDFS-7916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Rushabh S Shah
>Priority: Critical
> Fix For: 2.7.1
>
> Attachments: HDFS-7916-01.patch, HDFS-7916-1.patch
>
>
> if any badblock found, then BPSA for StandbyNode will go for infinite times
> to report it.
> {noformat}2015-03-11 19:43:41,528 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> stobdtserver3/10.224.54.70:18010
> org.apache.hadoop.hdfs.server.datanode.BPServiceActorActionException: Failed
> to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> at
> org.apache.hadoop.hdfs.server.datanode.ReportBadBlockAction.reportTo(ReportBadBlockAction.java:63)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processQueueMessages(BPServiceActor.java:1020)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:762)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:856)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9533) seen_txid in the shared edits directory is modified during bootstrapping
Kihwal Lee created HDFS-9533: Summary: seen_txid in the shared edits directory is modified during bootstrapping Key: HDFS-9533 URL: https://issues.apache.org/jira/browse/HDFS-9533 Project: Hadoop HDFS Issue Type: Bug Components: ha, namenode Affects Versions: 2.6.0 Reporter: Kihwal Lee The last known transaction id is stored in the seen_txid file of all known directories of a NNStorage when starting a new edit segment. However, we have seen a case where it contains an id that falls in the middle of an edit segment. This was the seen_txid file in the sahred edits directory. The active namenode's local storage was containing valid looking seen_txid. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9533) seen_txid in the shared edits directory is modified during bootstrapping
[
https://issues.apache.org/jira/browse/HDFS-9533?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049203#comment-15049203
]
Kihwal Lee commented on HDFS-9533:
--
It turned out a standby namenode was bootstrapped.
This is from {{BootstrapStandby#downloadImage()}} or {{doRun()}} in 2.6.
{code:java}
// 1
long curTxId = proxy.getTransactionID();
// 2
image.initEditLog(StartupOption.REGULAR);
// 3
image.getStorage().writeTransactionIdFileToStorage(curTxId);
{code}
(1) gets the current txid from the active node via rpc. (2) causes
{{editLog.initSharedJournalsForRead()}} to be called and the {{NNStorage}} will
contain the shared edits directory after that. When (3) is called, the txid
obtained in (1) will be written to the shared edits directory.
No matter what the intention of this code was, the shared edits directory
shouldn't be altered by non-active namenode.
> seen_txid in the shared edits directory is modified during bootstrapping
>
>
> Key: HDFS-9533
> URL: https://issues.apache.org/jira/browse/HDFS-9533
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, namenode
>Affects Versions: 2.6.0
>Reporter: Kihwal Lee
>
> The last known transaction id is stored in the seen_txid file of all known
> directories of a NNStorage when starting a new edit segment. However, we have
> seen a case where it contains an id that falls in the middle of an edit
> segment. This was the seen_txid file in the sahred edits directory. The
> active namenode's local storage was containing valid looking seen_txid.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9532) Detailed exception info is lost in reportTo method of ErrorReportAction and ReportBadBlockAction
[
https://issues.apache.org/jira/browse/HDFS-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-9532:
Priority: Trivial (was: Major)
> Detailed exception info is lost in reportTo method of ErrorReportAction and
> ReportBadBlockAction
>
>
> Key: HDFS-9532
> URL: https://issues.apache.org/jira/browse/HDFS-9532
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
>
> See code below
> {code}
> try {
> bpNamenode.reportBadBlocks(locatedBlock);
> } catch (RemoteException re) {
> DataNode.LOG.info("reportBadBlock encountered RemoteException for "
> + "block: " + block , re);
> } catch (IOException e) {
> throw new BPServiceActorActionException("Failed to report bad block "
> + block + " to namenode: ");
> }
> }
> {code}
> When IOException e is thrown, the information of e is not reported back to
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9532) Detailed exception info is lost in reportTo method of ErrorReportAction and ReportBadBlockAction
[
https://issues.apache.org/jira/browse/HDFS-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-9532:
Attachment: HDFS-9532.001.patch
Submitted patch 001. Thanks for reviewing.
> Detailed exception info is lost in reportTo method of ErrorReportAction and
> ReportBadBlockAction
>
>
> Key: HDFS-9532
> URL: https://issues.apache.org/jira/browse/HDFS-9532
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HDFS-9532.001.patch
>
>
> See code below
> {code}
> try {
> bpNamenode.reportBadBlocks(locatedBlock);
> } catch (RemoteException re) {
> DataNode.LOG.info("reportBadBlock encountered RemoteException for "
> + "block: " + block , re);
> } catch (IOException e) {
> throw new BPServiceActorActionException("Failed to report bad block "
> + block + " to namenode: ");
> }
> }
> {code}
> When IOException e is thrown, the information of e is not reported back to
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9532) Detailed exception info is lost in reportTo method of ErrorReportAction and ReportBadBlockAction
[
https://issues.apache.org/jira/browse/HDFS-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-9532:
Status: Patch Available (was: Open)
> Detailed exception info is lost in reportTo method of ErrorReportAction and
> ReportBadBlockAction
>
>
> Key: HDFS-9532
> URL: https://issues.apache.org/jira/browse/HDFS-9532
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>Priority: Trivial
> Labels: supportability
> Attachments: HDFS-9532.001.patch
>
>
> See code below
> {code}
> try {
> bpNamenode.reportBadBlocks(locatedBlock);
> } catch (RemoteException re) {
> DataNode.LOG.info("reportBadBlock encountered RemoteException for "
> + "block: " + block , re);
> } catch (IOException e) {
> throw new BPServiceActorActionException("Failed to report bad block "
> + block + " to namenode: ");
> }
> }
> {code}
> When IOException e is thrown, the information of e is not reported back to
> caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9445) Deadlock in datanode
[
https://issues.apache.org/jira/browse/HDFS-9445?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049119#comment-15049119
]
Chris Nauroth commented on HDFS-9445:
-
[~kihwal] and [~walter.k.su], if I understand correctly, this deadlock was
introduced by the HDFS-6788 patch (introduction of the read-write lock in
{{BPOfferService}}, and the deadlock can occur if there is concurrent heartbeat
processing while the disk checker thread is also taking a volume out of
service. Does that sound accurate?
> Deadlock in datanode
>
>
> Key: HDFS-9445
> URL: https://issues.apache.org/jira/browse/HDFS-9445
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.7.2
>Reporter: Kihwal Lee
>Assignee: Walter Su
>Priority: Blocker
> Attachments: HDFS-9445.00.patch, HDFS-9445.01.patch,
> HDFS-9445.02.patch
>
>
> {noformat}
> Found one Java-level deadlock:
> =
> "DataXceiver for client DFSClient_attempt_xxx at /1.2.3.4:100 [Sending block
> BP-x:blk_123_456]":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> "Thread-565":
> waiting for ownable synchronizer 0xd55613c8, (a
> java.util.concurrent.locks.ReentrantReadWriteLock$NonfairSync),
> which is held by "DataNode: heartbeating to my-nn:8020"
> "DataNode: heartbeating to my-nn:8020":
> waiting to lock monitor 0x7f77d0731768 (object 0xd60d9930, a
> org.apache.hadoop.hdfs.server.datanode.fsdataset.impl.FsDatasetImpl),
> which is held by "Thread-565"
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049177#comment-15049177 ] Bob Hansen commented on HDFS-9523: -- This passes fine on my local Docker instance with this patch (it wasn't before the patch). [~aw] - Is there any way to capture the output artifacts (notably the LastTest.log file) from CI runs, or is the volume too high on the Apache servers? I can start hacking the patches to dump more to the console, but getting the actual errors out would be a big help in tracking this down. > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-9532) Detailed exception info is lost in reportTo method of ErrorReportAction and ReportBadBlockAction
Yongjun Zhang created HDFS-9532:
---
Summary: Detailed exception info is lost in reportTo method of
ErrorReportAction and ReportBadBlockAction
Key: HDFS-9532
URL: https://issues.apache.org/jira/browse/HDFS-9532
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
See code below
{code}
try {
bpNamenode.reportBadBlocks(locatedBlock);
} catch (RemoteException re) {
DataNode.LOG.info("reportBadBlock encountered RemoteException for "
+ "block: " + block , re);
} catch (IOException e) {
throw new BPServiceActorActionException("Failed to report bad block "
+ block + " to namenode: ");
}
}
{code}
When IOException e is thrown, the information of e is not reported back to
caller.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.
[ https://issues.apache.org/jira/browse/HDFS-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-9483: - Attachment: HDFS-9483.001.patch Resubmitting patch to trigger jenkins. > Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured > WebHDFS. > - > > Key: HDFS-9483 > URL: https://issues.apache.org/jira/browse/HDFS-9483 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Reporter: Chris Nauroth >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9483.001.patch, HDFS-9483.patch > > > If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in > a URL to access it. The current documentation does not state this anywhere. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9526) DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore
[ https://issues.apache.org/jira/browse/HDFS-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049147#comment-15049147 ] Xiaobing Zhou commented on HDFS-9526: - Test failures are not related. Kindly commit it, thanks! > DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore > - > > Key: HDFS-9526 > URL: https://issues.apache.org/jira/browse/HDFS-9526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Attachments: HDFS-9526-HDFS-1312.001.patch > > > This is to > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnore > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnoreProperties > to > org.codehaus.jackson.annotate.JsonIgnore > org.codehaus.jackson.annotate.JsonIgnoreProperties, since > org.codehaus.jackson.map.ObjectMapper does not recognize the former when we > mark get/set accessors as @JsonIgnore. This will cause > UnrecognizedPropertyException for newly added accessors while doing Json > parsing using codehaus Jackson ObjectMapper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9483) Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured WebHDFS.
[
https://issues.apache.org/jira/browse/HDFS-9483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049149#comment-15049149
]
Hadoop QA commented on HDFS-9483:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 26s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 8s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} whitespace {color} | {color:red} 0m 0s
{color} | {color:red} The patch has 58 line(s) that end in whitespace. Use git
apply --whitespace=fix. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
37s {color} | {color:green} Patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 3m 27s {color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12776616/HDFS-9483.001.patch |
| JIRA Issue | HDFS-9483 |
| Optional Tests | asflicense mvnsite |
| uname | Linux 4fc5691467d1 3.13.0-36-lowlatency #63-Ubuntu SMP PREEMPT Wed
Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/hadoop/patchprocess/precommit/personality/provided.sh
|
| git revision | trunk / 50edcb9 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/13816/artifact/patchprocess/whitespace-eol.txt
|
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Max memory used | 32MB |
| Powered by | Apache Yetushttp://yetus.apache.org |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/13816/console |
This message was automatically generated.
> Documentation does not cover use of "swebhdfs" as URL scheme for SSL-secured
> WebHDFS.
> -
>
> Key: HDFS-9483
> URL: https://issues.apache.org/jira/browse/HDFS-9483
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Reporter: Chris Nauroth
>Assignee: Surendra Singh Lilhore
> Attachments: HDFS-9483.001.patch, HDFS-9483.patch
>
>
> If WebHDFS is secured with SSL, then you can use "swebhdfs" as the scheme in
> a URL to access it. The current documentation does not state this anywhere.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[ https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049398#comment-15049398 ] Tsz Wo Nicholas Sze commented on HDFS-9527: --- Thanks, Jing. Will fix TestClientProtocolForPipelineRecovery. On a second thought, we should change Namesystem.isInSnapshot to pass blockCollectionID and should not change getBlockCollection. Then, Namesystem no longer uses BlockInfo. > The return type of FSNamesystem.getBlockCollection should be changed to > INodeFile > - > > Key: HDFS-9527 > URL: https://issues.apache.org/jira/browse/HDFS-9527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h9527_20151208.patch > > > FSNamesystem.getBlockCollection always returns INodeFile. It avoids > unnecessary conversion from BlockCollection to INode/INodeFile after the > change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9526) DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore
[ https://issues.apache.org/jira/browse/HDFS-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaobing Zhou updated HDFS-9526: Description: This is to change org.apache.htrace.fasterxml.jackson.annotation.JsonIgnore org.apache.htrace.fasterxml.jackson.annotation.JsonIgnoreProperties to org.codehaus.jackson.annotate.JsonIgnore org.codehaus.jackson.annotate.JsonIgnoreProperties, since org.codehaus.jackson.map.ObjectMapper does not recognize the former when we mark get/set accessors as @JsonIgnore. This will cause UnrecognizedPropertyException for newly added accessors while doing Json parsing using codehaus Jackson ObjectMapper. was: This is to org.apache.htrace.fasterxml.jackson.annotation.JsonIgnore org.apache.htrace.fasterxml.jackson.annotation.JsonIgnoreProperties to org.codehaus.jackson.annotate.JsonIgnore org.codehaus.jackson.annotate.JsonIgnoreProperties, since org.codehaus.jackson.map.ObjectMapper does not recognize the former when we mark get/set accessors as @JsonIgnore. This will cause UnrecognizedPropertyException for newly added accessors while doing Json parsing using codehaus Jackson ObjectMapper. > DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore > - > > Key: HDFS-9526 > URL: https://issues.apache.org/jira/browse/HDFS-9526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: HDFS-1312 > > Attachments: HDFS-9526-HDFS-1312.001.patch > > > This is to change > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnore > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnoreProperties > to > org.codehaus.jackson.annotate.JsonIgnore > org.codehaus.jackson.annotate.JsonIgnoreProperties, since > org.codehaus.jackson.map.ObjectMapper does not recognize the former when we > mark get/set accessors as @JsonIgnore. This will cause > UnrecognizedPropertyException for newly added accessors while doing Json > parsing using codehaus Jackson ObjectMapper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[ https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9527: -- Attachment: h9527_20151209.patch Here is a new patch. h9527_20151209.patch > The return type of FSNamesystem.getBlockCollection should be changed to > INodeFile > - > > Key: HDFS-9527 > URL: https://issues.apache.org/jira/browse/HDFS-9527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Attachments: h9527_20151208.patch, h9527_20151209.patch > > > FSNamesystem.getBlockCollection always returns INodeFile. It avoids > unnecessary conversion from BlockCollection to INode/INodeFile after the > change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049494#comment-15049494 ] James Clampffer commented on HDFS-9523: --- [~bobthansen] - I was able to get the shim test to fail locally, I'm attaching the LastTest.log file. I'll leave that docker instance running in case you need anything else from it. > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch, failed_docker_run.txt > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9526) DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore
[ https://issues.apache.org/jira/browse/HDFS-9526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9526: -- Resolution: Fixed Fix Version/s: HDFS-1312 Status: Resolved (was: Patch Available) I have committed this. Thanks, Xiaobing! > DiskBalancer: change htrace...JsonIgnore to codehaus...JsonIgnore > - > > Key: HDFS-9526 > URL: https://issues.apache.org/jira/browse/HDFS-9526 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode >Affects Versions: 2.8.0 >Reporter: Xiaobing Zhou >Assignee: Xiaobing Zhou > Fix For: HDFS-1312 > > Attachments: HDFS-9526-HDFS-1312.001.patch > > > This is to > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnore > org.apache.htrace.fasterxml.jackson.annotation.JsonIgnoreProperties > to > org.codehaus.jackson.annotate.JsonIgnore > org.codehaus.jackson.annotate.JsonIgnoreProperties, since > org.codehaus.jackson.map.ObjectMapper does not recognize the former when we > mark get/set accessors as @JsonIgnore. This will cause > UnrecognizedPropertyException for newly added accessors while doing Json > parsing using codehaus Jackson ObjectMapper. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9411) HDFS ZoneLabel support
[ https://issues.apache.org/jira/browse/HDFS-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049509#comment-15049509 ] Zhe Zhang commented on HDFS-9411: - Thanks for the interesting proposal Vinay. Are the file/dir => zone assignments treated as hints or are they mandatory? If mandatory, we need to consider the scenarios where zones run out of space. [~mark] I'm a little confused by your comments above. It looks like a reply to some message that's not shown here. > HDFS ZoneLabel support > -- > > Key: HDFS-9411 > URL: https://issues.apache.org/jira/browse/HDFS-9411 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS_ZoneLabels-16112015.pdf > > > HDFS currently stores data blocks on different datanodes chosen by > BlockPlacement Policy. These datanodes are random within the > scope(local-rack/different-rack/nodegroup) of network topology. > In Multi-tenant (Tenant can be user/service) scenario, blocks of any tenant > can be on any datanodes. > Based on applications of different tenant, sometimes datanode might get busy > making the other tenant's application to slow down. It would be better if > admin's have a provision to logically divide the cluster among multi-tenants. > ZONE_LABELS can logically divide the cluster datanodes into multiple Zones. > High level design doc to follow soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9365) Balaner does not work with the HDFS-6376 HA setup
[ https://issues.apache.org/jira/browse/HDFS-9365?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049416#comment-15049416 ] Tsz Wo Nicholas Sze commented on HDFS-9365: --- Actually, do you mean combined keys DFSUtilClient.concatSuffixes(key, nsId)? The nsId is an internal ns after the patch. So I think it is fine. > Balaner does not work with the HDFS-6376 HA setup > - > > Key: HDFS-9365 > URL: https://issues.apache.org/jira/browse/HDFS-9365 > Project: Hadoop HDFS > Issue Type: Bug > Components: balancer & mover >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze > Attachments: h9365_20151119.patch, h9365_20151120.patch > > > HDFS-6376 added support for DistCp between two HA clusters. After the > change, Balaner will use all the NN from both the local and the remote > clusters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9530) huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
[
https://issues.apache.org/jira/browse/HDFS-9530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049367#comment-15049367
]
Chris Nauroth commented on HDFS-9530:
-
Additionally, Apache Hadoop 2.7.1 has a bug that configured
{{dfs.datanode.du.reserved}} space gets counted towards non-DFS used. The work
of fixing this is tracked in HDFS-9038.
> huge Non-DFS Used in hadoop 2.6.2 & 2.7.1
> -
>
> Key: HDFS-9530
> URL: https://issues.apache.org/jira/browse/HDFS-9530
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Fei Hui
>
> i run a hive job, and errors are as follow
> ===
> Diagnostic Messages for this Task:
> Error: java.lang.RuntimeException:
> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while
> processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:172)
> at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:450)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:163)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime
> Error while processing row {"k":"1","v":1}
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:518)
> at
> org.apache.hadoop.hive.ql.exec.mr.ExecMapper.map(ExecMapper.java:163)
> ... 8 more
> Caused by: org.apache.hadoop.hive.ql.metadata.HiveException:
> org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:663)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.addBlock(ClientNamenodeProtocolServerSideTranslatorPB.java:482)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:619)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:962)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2040)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2036)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1656)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2034)
> at
> org.apache.hadoop.hive.ql.exec.FileSinkOperator.process(FileSinkOperator.java:787)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.SelectOperator.process(SelectOperator.java:88)
> at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:837)
> at
> org.apache.hadoop.hive.ql.exec.TableScanOperator.process(TableScanOperator.java:97)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator$MapOpCtx.forward(MapOperator.java:162)
> at
> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:508)
> ... 9 more
> Caused by: org.apache.hadoop.ipc.RemoteException(java.io.IOException): File
> /test_abc/.hive-staging_hive_2015-12-09_15-24-10_553_7745334154733108653-1/_task_tmp.-ext-10002/pt=23/_tmp.17_3
> could only be replicated to 0 nodes instead of minReplication (=1). There
> are 3 datanode(s) running and no node(s) are excluded in this operation.
> at
> org.apache.hadoop.hdfs.server.blockmanagement.BlockManager.chooseTarget4NewBlock(BlockManager.java:1562)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:3245)
> at
>
[jira] [Updated] (HDFS-9523) libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail
[ https://issues.apache.org/jira/browse/HDFS-9523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Clampffer updated HDFS-9523: -- Attachment: failed_docker_run.txt > libhdfs++: failure to connect to ipv6 host causes CI unit tests to fail > --- > > Key: HDFS-9523 > URL: https://issues.apache.org/jira/browse/HDFS-9523 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: hdfs-client >Reporter: Bob Hansen >Assignee: Bob Hansen > Attachments: HDFS-9523.HDFS-8707.000.patch, failed_docker_run.txt > > > When run under Docker, libhdfs++ is not connecting to the mini DFS cluster. > This is the reason the CI tests have been failing in the > libhdfs_threaded_hdfspp_test_shim_static test. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9342) Erasure coding: client should update and commit block based on acknowledged size
[
https://issues.apache.org/jira/browse/HDFS-9342?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049649#comment-15049649
]
Zhe Zhang commented on HDFS-9342:
-
Thanks Walter (and sorry about the late review). The algorithm LGTM. A minor
below:
{code}
if (expected != acked) {
return ackedBGLength;
}
{code}
This isn't completely accurate: {{acked}} could be smaller than {{expected}} if
some packets are not acknowledged yet (not a DN failure). In that case I think
we should count {{acked}}. That said, the entire {{considerLastPartialStripe}}
logic is complex and not really necessary -- only used by an {{assert}}. Maybe
we should get rid of it?
> Erasure coding: client should update and commit block based on acknowledged
> size
>
>
> Key: HDFS-9342
> URL: https://issues.apache.org/jira/browse/HDFS-9342
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: erasure-coding
>Affects Versions: 3.0.0
>Reporter: Zhe Zhang
>Assignee: Walter Su
> Attachments: HDFS-9342.01.patch, HDFS-9342.02.patch
>
>
> For non-EC files, we have:
> {code}
> protected ExtendedBlock block; // its length is number of bytes acked
> {code}
> For EC files, the size of {{DFSStripedOutputStream#currentBlockGroup}} is
> incremented in {{writeChunk}} without waiting for ack. And both
> {{updatePipeline}} and {{commitBlock}} are based on size of
> {{currentBlockGroup}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049671#comment-15049671
]
Xiao Chen commented on HDFS-9519:
-
Thanks [~liuml07] and [~aw] for the comments and sorry for my delayed response.
The {{if (secondary != null)}} is confusing, and should be removed with the
change of HDFS-6348. My bad I didn't catch this redundancy when fixing
HDFS-3059.
The reason of disabling the http server when starting with {{CommandLineOpts}}
is that, after processing it the 2NN will terminate. E.g. when someone runs
{{hdfs secondarynamenode -checkpoint}} the 2nn just start and checkpoint, then
exit. Starting an http server during this is unnecessary - the 2nn webui is for
the daemon anyways. Also, when security is enabled, admin may not have the
credentials to the web server, but they should be allowed to do checkpointing.
My comments in the code seems to be doing more confusion than explanation..
I revised it and attached in patch 1 for review.
re. [~aw]'s questions:
bq. If the secondary namenode isn't actually configured but someone tries to
start the 2nn, what happens?
In this case 2NN is started as a daemon. My comments were bad.
bq. Also, does Checkpoint and Backup have different entry points or is this
used for those too?
AFAICT, 2NN only supports {{-checkpoint}}, {{-format}} and {{-geteditsize}},
with the same entry point. Backup should be done from the NN.
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9519:
Attachment: HDFS-9519.01.patch
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-9534) Add CLI command to clear storage policy from a path.
[
https://issues.apache.org/jira/browse/HDFS-9534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaobing Zhou reassigned HDFS-9534:
---
Assignee: Xiaobing Zhou
> Add CLI command to clear storage policy from a path.
>
>
> Key: HDFS-9534
> URL: https://issues.apache.org/jira/browse/HDFS-9534
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Reporter: Chris Nauroth
>Assignee: Xiaobing Zhou
>
> The {{hdfs storagepolicies}} command has sub-commands for
> {{-setStoragePolicy}} and {{-getStoragePolicy}} on a path. However, there is
> no {{-removeStoragePolicy}} to remove a previously set storage policy on a
> path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[
https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049657#comment-15049657
]
Hadoop QA commented on HDFS-9527:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 44s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 43s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
14s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
57s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 9s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 53s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
51s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 47s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m 45s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 17s
{color} | {color:red} Patch generated 3 new checkstyle issues in
hadoop-hdfs-project/hadoop-hdfs (total was 472, now 472). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m 56s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
19s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 12s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 57m 14s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 56m 28s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 142m 49s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.TestCrcCorruption |
| | hadoop.hdfs.server.namenode.TestFileTruncate |
| JDK v1.7.0_91 Failed junit tests | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestDFSStripedOutputStreamWithFailure060 |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12776636/h9527_20151209.patch |
| JIRA Issue | HDFS-9527 |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit findbugs
[jira] [Created] (HDFS-9535) Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
Jing Zhao created HDFS-9535:
---
Summary: Fix
TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
Key: HDFS-9535
URL: https://issues.apache.org/jira/browse/HDFS-9535
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.8.0
Reporter: Jing Zhao
Priority: Minor
TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate failed in several
Jenkins run (e.g.,
https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
failure message:
{code}
java.lang.AssertionError: Bad value for metric BlocksReplicated
Expected :0
Actual :1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
at
org.apache.hadoop.hdfs.TestReplication.assertNoReplicationWasPerformed(TestReplication.java:695)
at
org.apache.hadoop.hdfs.TestReplication.testNoExtraReplicationWhenBlockReceivedIsLate(TestReplication.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9535) Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
[
https://issues.apache.org/jira/browse/HDFS-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9535:
Assignee: Mingliang Liu
> Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
> -
>
> Key: HDFS-9535
> URL: https://issues.apache.org/jira/browse/HDFS-9535
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jing Zhao
>Assignee: Mingliang Liu
>Priority: Minor
>
> TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate failed in
> several Jenkins run (e.g.,
> https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
> failure is on the last {{assertNoReplicationWasPerformed}} check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9535) Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
[
https://issues.apache.org/jira/browse/HDFS-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049755#comment-15049755
]
Jing Zhao commented on HDFS-9535:
-
The failure is because:
# In HDFS-1172 we update the pending replication queue if a block has minimum
required number of replicas but the NN has not received block-received report
from some DN
# The test writes two blocks into a file. For the first block, it is possible
that NN receives 0 block-received msg from DN but still commit the block when
the client tries to get the next block. In that case we will add the block into
under-replicated queue instead of the pending queue, and block recovery will
happen in the cluster.
# This usually will not happen when closing the file since the client will
retry the completeFile call until the minimum replication is satisfied. Thus a
simple fix can be changing the file length to 1 block.
> Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
> -
>
> Key: HDFS-9535
> URL: https://issues.apache.org/jira/browse/HDFS-9535
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jing Zhao
>Priority: Minor
>
> TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate failed in
> several Jenkins run (e.g.,
> https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
> failure is on the last {{assertNoReplicationWasPerformed}} check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[ https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-9527: -- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 2.8.0 Status: Resolved (was: Patch Available) I have committed this. > The return type of FSNamesystem.getBlockCollection should be changed to > INodeFile > - > > Key: HDFS-9527 > URL: https://issues.apache.org/jira/browse/HDFS-9527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.8.0 > > Attachments: h9527_20151208.patch, h9527_20151209.patch > > > FSNamesystem.getBlockCollection always returns INodeFile. It avoids > unnecessary conversion from BlockCollection to INode/INodeFile after the > change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9532) Detailed exception info is lost in reportTo method of ErrorReportAction and ReportBadBlockAction
[
https://issues.apache.org/jira/browse/HDFS-9532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049598#comment-15049598
]
Hadoop QA commented on HDFS-9532:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m 0s
{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 11m
35s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 17s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
19s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
45s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 55s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 49s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
15s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m 7s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 21s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
18s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m 9s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 54s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 103m 17s
{color} | {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66.
{color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 90m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 56 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 235m 24s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.net.TestNetworkTopology |
| | hadoop.hdfs.TestQuota |
| | hadoop.hdfs.server.namenode.ha.TestEditLogTailer |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.TestReadStripedFileWithDecoding |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailure |
| | hadoop.hdfs.server.datanode.TestBlockReplacement |
| | hadoop.hdfs.server.namenode.ha.TestSeveralNameNodes |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.hdfs.TestFileCreationClient |
| |
[jira] [Commented] (HDFS-9516) truncate file fails with data dirs on multiple disks
[
https://issues.apache.org/jira/browse/HDFS-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049658#comment-15049658
]
Konstantin Shvachko commented on HDFS-9516:
---
Btw which branch are you running it on? Couldn't match exactly the line numbers
with any of the 2s.
> truncate file fails with data dirs on multiple disks
>
>
> Key: HDFS-9516
> URL: https://issues.apache.org/jira/browse/HDFS-9516
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Bogdan Raducanu
> Attachments: Main.java, truncate.dn.log
>
>
> FileSystem.truncate returns false (no exception) but the file is never closed
> and not writable after this.
> It seems to be because of copy on truncate which is used because the system
> is in upgrade state. In this case a rename between devices is attempted.
> See attached log and repro code.
> Probably also affects truncate snapshotted file when copy on truncate is also
> used.
> Possibly it affects not only truncate but any block recovery.
> I think the problem is in updateReplicaUnderRecovery
> {code}
> ReplicaBeingWritten newReplicaInfo = new ReplicaBeingWritten(
> newBlockId, recoveryId, rur.getVolume(),
> blockFile.getParentFile(),
> newlength);
> {code}
> blockFile is created with copyReplicaWithNewBlockIdAndGS which is allowed to
> choose any volume so rur.getVolume() is not where the block is located.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9034) "StorageTypeStats" Metric should not count failed storage.
[ https://issues.apache.org/jira/browse/HDFS-9034?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049781#comment-15049781 ] Benoy Antony commented on HDFS-9034: I'll review this tomorrow. > "StorageTypeStats" Metric should not count failed storage. > -- > > Key: HDFS-9034 > URL: https://issues.apache.org/jira/browse/HDFS-9034 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Archana T >Assignee: Surendra Singh Lilhore > Attachments: HDFS-9034.01.patch, HDFS-9034.02.patch, > HDFS-9034.03.patch, HDFS-9034.04.patch, dfsStorage_NN_UI2.png > > > When we remove one storage type from all the DNs, still NN UI shows entry of > those storage type -- > Ex:for ARCHIVE > Steps-- > 1. ARCHIVE Storage type was added for all DNs > 2. Stop DNs > 3. Removed ARCHIVE Storages from all DNs > 4. Restarted DNs > NN UI shows below -- > DFS Storage Types > Storage Type Configured Capacity Capacity Used Capacity Remaining > ARCHIVE 57.18 GB64 KB (0%) 39.82 GB (69.64%) 64 KB > 1 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049820#comment-15049820
]
Tsz Wo Nicholas Sze commented on HDFS-8578:
---
There are two different problems:
# data dirs are processed sequentially;
# OOM if all data dirs are processed in parallel.
I suggest that we fix #1 first and then fix #2 in a separated JIRA in order to
keep the patch simple. Also, #2 need more discussion. We should also have
numParallelThreads configurable for the moment due to #2. At least, users
could set it to numParallelThreads to 1 to avoid #2. What do you think?
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049855#comment-15049855
]
Yongjun Zhang commented on HDFS-9519:
-
Thanks [~xiaochen] for the patch.
Can we move the main comment to the beginning of the block, something like:
{code}
// SecondaryNameNode can be started to run a command (i.e. checkpoint or
// geteditsize etc) and terminate, or to run as a daemon when no command
is
// specified. The web server is only needed when 2NN runs as a daemon.
if (opts != null && opts.getCommand() != null) {
// 2NN is started to run a command and then terminate
int ret = secondary.processStartupCommand(opts);
terminate(ret);
}
// 2NN runs as a daemon, start the web server
secondary.startInfoServer();
{code}
Thanks.
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9038) Reserved space is erroneously counted towards non-DFS used.
[
https://issues.apache.org/jira/browse/HDFS-9038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049705#comment-15049705
]
Chris Nauroth commented on HDFS-9038:
-
I'm in favor of using {{File#getUsableSpace}} for the sake of consistency with
the pre-HDFS-5215 behavior. I'd also like to make sure Arpit gets a chance to
respond too, but I don't expect he'll have a chance to comment until next week.
> Reserved space is erroneously counted towards non-DFS used.
> ---
>
> Key: HDFS-9038
> URL: https://issues.apache.org/jira/browse/HDFS-9038
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Chris Nauroth
>Assignee: Brahma Reddy Battula
> Attachments: HDFS-9038-002.patch, HDFS-9038-003.patch,
> HDFS-9038-004.patch, HDFS-9038-005.patch, HDFS-9038.patch
>
>
> HDFS-5215 changed the DataNode volume available space calculation to consider
> the reserved space held by the {{dfs.datanode.du.reserved}} configuration
> property. As a side effect, reserved space is now counted towards non-DFS
> used. I don't believe it was intentional to change the definition of non-DFS
> used. This issue proposes restoring the prior behavior: do not count
> reserved space towards non-DFS used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049840#comment-15049840
]
Tsz Wo Nicholas Sze commented on HDFS-8578:
---
> ... Also, #2 need more discussion. ...
Datanode's memory should be large enough to hold all the blocks, otherwise, it
won't run. So I think using BlockingQueue with a fixed capacity may not be the
best way to solve the problem. We should be able to reduce the memory
footprint in the data structures.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-9534) Add CLI command to clear storage policy from a path.
Chris Nauroth created HDFS-9534:
---
Summary: Add CLI command to clear storage policy from a path.
Key: HDFS-9534
URL: https://issues.apache.org/jira/browse/HDFS-9534
Project: Hadoop HDFS
Issue Type: Improvement
Components: tools
Reporter: Chris Nauroth
The {{hdfs storagepolicies}} command has sub-commands for {{-setStoragePolicy}}
and {{-getStoragePolicy}} on a path. However, there is no
{{-removeStoragePolicy}} to remove a previously set storage policy on a path.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9173) Erasure Coding: Lease recovery for striped file
[
https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9173:
Attachment: HDFS-9173.06.patch
Upload a rebased patch for Walter. Also did some minor code cleanup in class
{{RecoveryTaskStriped}} by removing some redundant class member fields.
bq. better to use if-else to avoid constructing RecoveringBlock unnecessarily
The original patch only defines a copy constructor for
{{RecoveryStripedBlock}}, thus in that piece of code it may be hard to use a
if-else unless we change the constructor definition. Maybe it's ok now just
leaving it there?
> Erasure Coding: Lease recovery for striped file
> ---
>
> Key: HDFS-9173
> URL: https://issues.apache.org/jira/browse/HDFS-9173
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Walter Su
>Assignee: Walter Su
> Attachments: HDFS-9173.00.wip.patch, HDFS-9173.01.patch,
> HDFS-9173.02.step125.patch, HDFS-9173.03.patch, HDFS-9173.04.patch,
> HDFS-9173.05.patch, HDFS-9173.06.patch
>
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9516) truncate file fails with data dirs on multiple disks
[
https://issues.apache.org/jira/browse/HDFS-9516?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049644#comment-15049644
]
Konstantin Shvachko commented on HDFS-9516:
---
Indeed, looks like an attempt to rename across volumes. Good catch, Bogdan. And
analysis too.
The problem is that {{copyReplicaWithNewBlockIdAndGS()}} does not take into
account which volume is the {{rur}} replica on, and can choose a different one.
I don't think this affects anything, but truncate in the case of
copy-on-truncate, which involves upgrades and snapshots.
I was wondering if you traced this condition further in time. This recovery
should fail, and another would start some time later, eventually the same
volume should be chosen and that last recovery should succeed.
> truncate file fails with data dirs on multiple disks
>
>
> Key: HDFS-9516
> URL: https://issues.apache.org/jira/browse/HDFS-9516
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.1
>Reporter: Bogdan Raducanu
> Attachments: Main.java, truncate.dn.log
>
>
> FileSystem.truncate returns false (no exception) but the file is never closed
> and not writable after this.
> It seems to be because of copy on truncate which is used because the system
> is in upgrade state. In this case a rename between devices is attempted.
> See attached log and repro code.
> Probably also affects truncate snapshotted file when copy on truncate is also
> used.
> Possibly it affects not only truncate but any block recovery.
> I think the problem is in updateReplicaUnderRecovery
> {code}
> ReplicaBeingWritten newReplicaInfo = new ReplicaBeingWritten(
> newBlockId, recoveryId, rur.getVolume(),
> blockFile.getParentFile(),
> newlength);
> {code}
> blockFile is created with copyReplicaWithNewBlockIdAndGS which is allowed to
> choose any volume so rur.getVolume() is not where the block is located.
>
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9514) TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; exception being swallowed
[
https://issues.apache.org/jira/browse/HDFS-9514?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049709#comment-15049709
]
Yongjun Zhang commented on HDFS-9514:
-
Thanks [~ste...@apache.org] for reporting the issue and [~jojochuang] for the
patch.
One comment, the following block (two occurences):
{code}
catch (Throwable t) {
Assert.fail("wrong exception:" + t.toString() +
ExceptionUtils.getStackTrace(t));
}
{code}
can be dropped, since the test itself would print out the exception info
including the stack.
Thanks.
> TestDistributedFileSystem.testDFSClientPeerWriteTimeout failing; exception
> being swallowed
> --
>
> Key: HDFS-9514
> URL: https://issues.apache.org/jira/browse/HDFS-9514
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, test
>Affects Versions: 3.0.0
> Environment: jenkins
>Reporter: Steve Loughran
>Assignee: Wei-Chiu Chuang
> Attachments: HDFS-9514.001.patch, HDFS-9514.002.patch
>
>
> {{TestDistributedFileSystem.testDFSClientPeerWriteTimeout}} is failing with
> the wrong exception being raised...reporter isn't using the
> {{GenericTestUtils}} code and so losing the details
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[
https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049734#comment-15049734
]
Jing Zhao commented on HDFS-9527:
-
The new patch looks good to me. The failed tests should be unrelated. +1
The failed {{TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate}}
seems flaky. We can fix it in a separate jira.
> The return type of FSNamesystem.getBlockCollection should be changed to
> INodeFile
> -
>
> Key: HDFS-9527
> URL: https://issues.apache.org/jira/browse/HDFS-9527
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h9527_20151208.patch, h9527_20151209.patch
>
>
> FSNamesystem.getBlockCollection always returns INodeFile. It avoids
> unnecessary conversion from BlockCollection to INode/INodeFile after the
> change.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9535) Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
[
https://issues.apache.org/jira/browse/HDFS-9535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-9535:
Description: TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
failed in several Jenkins run (e.g.,
https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
failure is on the last {{assertNoReplicationWasPerformed}} check. (was:
TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate failed in several
Jenkins run (e.g.,
https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
failure message:
{code}
java.lang.AssertionError: Bad value for metric BlocksReplicated
Expected :0
Actual :1
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at
org.apache.hadoop.test.MetricsAsserts.assertCounter(MetricsAsserts.java:228)
at
org.apache.hadoop.hdfs.TestReplication.assertNoReplicationWasPerformed(TestReplication.java:695)
at
org.apache.hadoop.hdfs.TestReplication.testNoExtraReplicationWhenBlockReceivedIsLate(TestReplication.java:615)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
at
org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
at
org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
{code})
> Fix TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate
> -
>
> Key: HDFS-9535
> URL: https://issues.apache.org/jira/browse/HDFS-9535
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.8.0
>Reporter: Jing Zhao
>Priority: Minor
>
> TestReplication#testNoExtraReplicationWhenBlockReceivedIsLate failed in
> several Jenkins run (e.g.,
> https://builds.apache.org/job/PreCommit-HDFS-Build/13818/testReport/). The
> failure is on the last {{assertNoReplicationWasPerformed}} check.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049978#comment-15049978
]
Xiao Chen commented on HDFS-9519:
-
Thanks Yongjun. I agree with your general idea that we should focus on the 2
modes instead of the web server itself. Quick clarification:
'Run as a daemon' is not limited to 'no command is specified' - if {{-format}}
is given, after {{parseArgs}}, {{opts.getCommand()}} still returns null so that
2NN is still started as a daemon.
Uploaded patch 2 which I think explains it clearer. Please let me know what you
think.
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch, HDFS-9519.02.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050038#comment-15050038
]
Vinayakumar B commented on HDFS-8578:
-
bq. I suggest that we fix #1 first and then fix #2 in a separated JIRA in order
to keep the patch simple. Also, #2 need more discussion. We should also have
numParallelThreads configurable for the moment due to #2. At least, users could
set it to numParallelThreads to 1 to avoid #2. What do you think?
Yes, we can take solving OOM problem in separate jira, just not to hurry.
I will update the patch for #1, along with configuration for numParallelThreads.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-branch-2.6.0.patch, HDFS-8578-branch-2.7-001.patch,
> HDFS-8578-branch-2.7-002.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9411) HDFS ZoneLabel support
[ https://issues.apache.org/jira/browse/HDFS-9411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050090#comment-15050090 ] Vinayakumar B commented on HDFS-9411: - Thanks [~zhz] for taking look here. bq. Are the file/dir => zone assignments treated as hints or are they mandatory? If mandatory, we need to consider the scenarios where zones run out of space. These are mandatory for files which have zones. Yes we need to consider those cases. I have plan to update the design in sometime. That will have these cases handled. > HDFS ZoneLabel support > -- > > Key: HDFS-9411 > URL: https://issues.apache.org/jira/browse/HDFS-9411 > Project: Hadoop HDFS > Issue Type: New Feature >Reporter: Vinayakumar B >Assignee: Vinayakumar B > Attachments: HDFS_ZoneLabels-16112015.pdf > > > HDFS currently stores data blocks on different datanodes chosen by > BlockPlacement Policy. These datanodes are random within the > scope(local-rack/different-rack/nodegroup) of network topology. > In Multi-tenant (Tenant can be user/service) scenario, blocks of any tenant > can be on any datanodes. > Based on applications of different tenant, sometimes datanode might get busy > making the other tenant's application to slow down. It would be better if > admin's have a provision to logically divide the cluster among multi-tenants. > ZONE_LABELS can logically divide the cluster datanodes into multiple Zones. > High level design doc to follow soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050172#comment-15050172
]
Yongjun Zhang commented on HDFS-9519:
-
Thanks Xiao. +1 on rev2 pending jenkins test.
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch, HDFS-9519.02.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9519:
Status: Patch Available (was: Open)
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch, HDFS-9519.02.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9173) Erasure Coding: Lease recovery for striped file
[
https://issues.apache.org/jira/browse/HDFS-9173?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049967#comment-15049967
]
Hadoop QA commented on HDFS-9173:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 1s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 7 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 10m
12s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 31s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 3s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
29s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
33s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m 3s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 10s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 1s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
49s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 2m 0s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 30s
{color} | {color:red} Patch generated 14 new checkstyle issues in
hadoop-hdfs-project (total was 599, now 606). {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 55s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
31s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 2m 45s
{color} | {color:red} hadoop-hdfs-project/hadoop-hdfs introduced 1 new FindBugs
issues. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m 7s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 3m 0s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 18s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 73m 16s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m 16s
{color} | {color:green} hadoop-hdfs-client in the patch passed with JDK
v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 70m 3s {color}
| {color:red} hadoop-hdfs in the patch failed with JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 32s
{color} | {color:red} Patch generated 58 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 198m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason ||
[jira] [Created] (HDFS-9536) OOM errors during parallel upgrade to Block-ID based layout
Vinayakumar B created HDFS-9536: --- Summary: OOM errors during parallel upgrade to Block-ID based layout Key: HDFS-9536 URL: https://issues.apache.org/jira/browse/HDFS-9536 Project: Hadoop HDFS Issue Type: Bug Reporter: Vinayakumar B Assignee: Vinayakumar B This is a follow-up jira for the OOM errors observed during parallel upgrade to Block-ID based datanode layout using HDFS-8578 fix. more clue [here|https://issues.apache.org/jira/browse/HDFS-8578?focusedCommentId=15042012=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-15042012] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-9529) Extend Erasure Code to support POWER Chip acceleration
[ https://issues.apache.org/jira/browse/HDFS-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang resolved HDFS-9529. - Resolution: Duplicate Thanks Qijun for confirming this. Resolving this one as dup. > Extend Erasure Code to support POWER Chip acceleration > -- > > Key: HDFS-9529 > URL: https://issues.apache.org/jira/browse/HDFS-9529 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: wqijun >Assignee: wqijun > Fix For: 3.0.0 > > > Erasure Code is a very important feature in new HDFS version. This JIRA will > focus on how to extend EC to support multiple types of EC acceleration by C > library and other hardware method, like GPU or FPGA. Compared with > Hadoop-11887, this JIRA will more focus on how to leverage POWER Chip > capability to accelerate the EC calculating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050067#comment-15050067
]
Vinayakumar B commented on HDFS-8578:
-
HDFS-9536 has been created to follow-up #2, OOM issue during parallel upgrade.
Lets take further discussion about that OOM there.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-15.patch, HDFS-8578-branch-2.6.0.patch,
> HDFS-8578-branch-2.7-001.patch, HDFS-8578-branch-2.7-002.patch,
> HDFS-8578-branch-2.7-003.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7916) 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop
[
https://issues.apache.org/jira/browse/HDFS-7916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050241#comment-15050241
]
Yongjun Zhang commented on HDFS-7916:
-
Hi [~vinayrpet],
Thanks for your reply. My question was, this jira suggest to ignore
StandbyException, but the fix only check RemoteException. RemoteException could
wrap exceptions other than StandbyException. Should we treat the other kind of
exceptions wrapped by RemoveException the same as StandbyException (the current
fix does that), or we should do more checking to handle StandbyException only
this way? Thanks.
> 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for
> infinite loop
> --
>
> Key: HDFS-7916
> URL: https://issues.apache.org/jira/browse/HDFS-7916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Rushabh S Shah
>Priority: Critical
> Fix For: 2.7.1
>
> Attachments: HDFS-7916-01.patch, HDFS-7916-1.patch
>
>
> if any badblock found, then BPSA for StandbyNode will go for infinite times
> to report it.
> {noformat}2015-03-11 19:43:41,528 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> stobdtserver3/10.224.54.70:18010
> org.apache.hadoop.hdfs.server.datanode.BPServiceActorActionException: Failed
> to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> at
> org.apache.hadoop.hdfs.server.datanode.ReportBadBlockAction.reportTo(ReportBadBlockAction.java:63)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processQueueMessages(BPServiceActor.java:1020)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:762)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:856)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9522) Cleanup o.a.h.hdfs.protocol.SnapshotDiffReport$DiffReportEntry
[ https://issues.apache.org/jira/browse/HDFS-9522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050239#comment-15050239 ] John Zhuge commented on HDFS-9522: -- Thanks. > Cleanup o.a.h.hdfs.protocol.SnapshotDiffReport$DiffReportEntry > -- > > Key: HDFS-9522 > URL: https://issues.apache.org/jira/browse/HDFS-9522 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: John Zhuge >Assignee: John Zhuge >Priority: Minor > Original Estimate: 24h > Remaining Estimate: 24h > > The current DiffReportEntry is a C-style tagged union-like data structure. > Recommend subclass hierarchy as in Java idiom. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9528) Cleanup namenode audit/log/exception messages
[
https://issues.apache.org/jira/browse/HDFS-9528?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049896#comment-15049896
]
Uma Maheswara Rao G commented on HDFS-9528:
---
Thankyou for the cleanup.
Seems test failures are related. Most of the failures are due to
{noformat}
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.toLeaseExceptionMessage(FSNamesystem.java:2503)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.checkLease(FSNamesystem.java:2511)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.analyzeFileState(FSDirWriteFileOp.java:652)
at
org.apache.hadoop.hdfs.server.namenode.FSDirWriteFileOp.validateAddBlock(FSDirWriteFileOp.java:185)
at
org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getAdditionalBlock(FSNamesystem.java:2383)
at
org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.addBlock(NameNodeRpcServer.java:796)
{noformat}
Here the reason is:
{code}
return src + " (inode " + fileId + ") " + lease != null ?
lease.toString(): "Holder " + holder + " does not have any open files.";
{code}
Brackets missed in above line. Here we should keep " condition? --- : --- "
part into brackets. Otherwise it is including prepended string also for not
equal evaluation as the precedence and always its a non null.
I think findbug warning is about the same too.
Also please check checkstyle warnings about max line length which is due to one
non formatted line.
Other than this patch looks good.
> Cleanup namenode audit/log/exception messages
> -
>
> Key: HDFS-9528
> URL: https://issues.apache.org/jira/browse/HDFS-9528
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Reporter: Tsz Wo Nicholas Sze
>Assignee: Tsz Wo Nicholas Sze
>Priority: Minor
> Attachments: h9528_20151208.patch
>
>
> - Cleanup unnecessary long methods for constructing message strings.
> - Avoid calling toString() methods.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9513) DataNodeManager#getDataNodeStorageInfos not backward compatibility
[
https://issues.apache.org/jira/browse/HDFS-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049938#comment-15049938
]
邓飞 commented on HDFS-9513:
--
yes, i will work on it,later upload new patch
> DataNodeManager#getDataNodeStorageInfos not backward compatibility
> --
>
> Key: HDFS-9513
> URL: https://issues.apache.org/jira/browse/HDFS-9513
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.2.0, 2.7.1
> Environment: 2.2.0 HDFS Client &2.7.1 HDFS Cluster
>Reporter: 邓飞
>Assignee: 邓飞
>Priority: Blocker
> Attachments: patch.HDFS-9513.20151207
>
>
> We is upgraded our new HDFS cluster to 2.7.1,but we YARN cluster is
> 2.2.0(8000+,it's too hard to upgrade as soon as HDFS cluster).
> The compatible case happened datasteamer do pipeline recovery, the NN need
> DN's storageInfo to update pipeline, and the storageIds is pair of
> pipleline's DN,but HDFS support storage type feature from 2.3.0
> [HDFS-2832|https://issues.apache.org/jira/browse/HDFS-2832], older version
> not have storageId ,although the protobuf serialization make the protocol
> compatible,but the client will throw remote exception as
> ArrayIndexOutOfBoundsException.
>
> the exception stack is below:
> {noformat}
> 2015-12-05 20:26:38,291 ERROR [Thread-4] org.apache.hadoop.hdfs.DFSClient:
> Failed to close file XXX
> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
> 0
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:513)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipelineInternal(FSNamesystem.java:6439)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipeline(FSNamesystem.java:6404)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updatePipeline(NameNodeRpcServer.java:892)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updatePipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:997)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1066)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updatePipeline(ClientNamenodeProtocolTranslatorPB.java:801)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy11.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1047)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7661) Support read when a EC file is being written
[
https://issues.apache.org/jira/browse/HDFS-7661?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049962#comment-15049962
]
GAO Rui commented on HDFS-7661:
---
[~szetszwo], [~jingzhao], thank you very much for the enlightening discussion
in the video meeting. I have walked through EC file reading part source codes.
In DFSInputStream#getFileLength():
{code}
public long getFileLength() {
synchronized(infoLock) {
return locatedBlocks == null? 0:
locatedBlocks.getFileLength() + lastBlockBeingWrittenLength;
}
}
{code}
I have three questions.
The first one, for a being written EC file, we should make
{{locatedBlocks.getFileLength()}} cover to the last completed block group,
right?
The second questions about {{lastBlockBeingWrittenLength}}.
I think for EC files, {{lastBlockBeingWrittenLength}} should be incremented to
the last completed written stripe. By completed written stripe(in R-S-6-3), I
refer to the stripe which has all internal cells(6 data cells and 3 parity
cells) written. According to the current writing part code. StripedDataStreamer
wait for acks when a stripe has all internal data cells full and parity cells
calculated. So, it is OK to keep incrementing {{lastBlockBeingWrittenLength}}
to the last completed written strip. Does it make sense to you?
The last question is about updating {{lastBlockBeingWrittenLength}} when
hflush/hsync is invoked. I would upload an document and try to cover all
possible scenarios in the document.
I have tried to trace {{lastBlockBeingWrittenLength}}, and found out that we
get the value of {{lastBlockBeingWrittenLength}} form the datanode side by
ReplicaBeingWritten#getVisibleLength():
{code}
@Override
public long getVisibleLength() {
return getBytesAcked(); // all acked bytes are visible
}
{code}
For EC files, it’s not appropriate to just take BytesAcked as visible length,
in the scenarios with flush/sync involved. I would over ride this method in the
document, too.
> Support read when a EC file is being written
>
>
> Key: HDFS-7661
> URL: https://issues.apache.org/jira/browse/HDFS-7661
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Tsz Wo Nicholas Sze
>Assignee: GAO Rui
> Attachments: EC-file-flush-and-sync-steps-plan-2015-12-01.png,
> HDFS-7661-unitTest-wip-trunk.patch
>
>
> We also need to support hflush/hsync and visible length.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9513) DataNodeManager#getDataNodeStorageInfos not backward compatibility
[
https://issues.apache.org/jira/browse/HDFS-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049937#comment-15049937
]
邓飞 commented on HDFS-9513:
--
yes, i will work on it,later upload new patch
> DataNodeManager#getDataNodeStorageInfos not backward compatibility
> --
>
> Key: HDFS-9513
> URL: https://issues.apache.org/jira/browse/HDFS-9513
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.2.0, 2.7.1
> Environment: 2.2.0 HDFS Client &2.7.1 HDFS Cluster
>Reporter: 邓飞
>Assignee: 邓飞
>Priority: Blocker
> Attachments: patch.HDFS-9513.20151207
>
>
> We is upgraded our new HDFS cluster to 2.7.1,but we YARN cluster is
> 2.2.0(8000+,it's too hard to upgrade as soon as HDFS cluster).
> The compatible case happened datasteamer do pipeline recovery, the NN need
> DN's storageInfo to update pipeline, and the storageIds is pair of
> pipleline's DN,but HDFS support storage type feature from 2.3.0
> [HDFS-2832|https://issues.apache.org/jira/browse/HDFS-2832], older version
> not have storageId ,although the protobuf serialization make the protocol
> compatible,but the client will throw remote exception as
> ArrayIndexOutOfBoundsException.
>
> the exception stack is below:
> {noformat}
> 2015-12-05 20:26:38,291 ERROR [Thread-4] org.apache.hadoop.hdfs.DFSClient:
> Failed to close file XXX
> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
> 0
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:513)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipelineInternal(FSNamesystem.java:6439)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipeline(FSNamesystem.java:6404)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updatePipeline(NameNodeRpcServer.java:892)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updatePipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:997)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1066)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updatePipeline(ClientNamenodeProtocolTranslatorPB.java:801)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy11.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1047)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9513) DataNodeManager#getDataNodeStorageInfos not backward compatibility
[
https://issues.apache.org/jira/browse/HDFS-9513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049936#comment-15049936
]
邓飞 commented on HDFS-9513:
--
yes, i will work on it,later upload new patch
> DataNodeManager#getDataNodeStorageInfos not backward compatibility
> --
>
> Key: HDFS-9513
> URL: https://issues.apache.org/jira/browse/HDFS-9513
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, namenode
>Affects Versions: 2.2.0, 2.7.1
> Environment: 2.2.0 HDFS Client &2.7.1 HDFS Cluster
>Reporter: 邓飞
>Assignee: 邓飞
>Priority: Blocker
> Attachments: patch.HDFS-9513.20151207
>
>
> We is upgraded our new HDFS cluster to 2.7.1,but we YARN cluster is
> 2.2.0(8000+,it's too hard to upgrade as soon as HDFS cluster).
> The compatible case happened datasteamer do pipeline recovery, the NN need
> DN's storageInfo to update pipeline, and the storageIds is pair of
> pipleline's DN,but HDFS support storage type feature from 2.3.0
> [HDFS-2832|https://issues.apache.org/jira/browse/HDFS-2832], older version
> not have storageId ,although the protobuf serialization make the protocol
> compatible,but the client will throw remote exception as
> ArrayIndexOutOfBoundsException.
>
> the exception stack is below:
> {noformat}
> 2015-12-05 20:26:38,291 ERROR [Thread-4] org.apache.hadoop.hdfs.DFSClient:
> Failed to close file XXX
> org.apache.hadoop.ipc.RemoteException(java.lang.ArrayIndexOutOfBoundsException):
> 0
> at
> org.apache.hadoop.hdfs.server.blockmanagement.DatanodeManager.getDatanodeStorageInfos(DatanodeManager.java:513)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipelineInternal(FSNamesystem.java:6439)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.updatePipeline(FSNamesystem.java:6404)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.updatePipeline(NameNodeRpcServer.java:892)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.updatePipeline(ClientNamenodeProtocolServerSideTranslatorPB.java:997)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:616)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1066)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2049)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:2045)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1844)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2043)
> at org.apache.hadoop.ipc.Client.call(Client.java:1347)
> at org.apache.hadoop.ipc.Client.call(Client.java:1300)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy10.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolTranslatorPB.updatePipeline(ClientNamenodeProtocolTranslatorPB.java:801)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:186)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:102)
> at com.sun.proxy.$Proxy11.updatePipeline(Unknown Source)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.setupPipelineForAppendOrRecovery(DFSOutputStream.java:1047)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.processDatanodeError(DFSOutputStream.java:823)
> at
> org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:475)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[ https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049949#comment-15049949 ] Hudson commented on HDFS-9527: -- FAILURE: Integrated in Hadoop-trunk-Commit #8949 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/8949/]) HDFS-9527. The return type of FSNamesystem.getBlockCollection should be (szetszwo: rev 132478e805ba0f955345217b8ad87c2d17cccb2d) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > The return type of FSNamesystem.getBlockCollection should be changed to > INodeFile > - > > Key: HDFS-9527 > URL: https://issues.apache.org/jira/browse/HDFS-9527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.8.0 > > Attachments: h9527_20151208.patch, h9527_20151209.patch > > > FSNamesystem.getBlockCollection always returns INodeFile. It avoids > unnecessary conversion from BlockCollection to INode/INodeFile after the > change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-9529) Extend Erasure Code to support POWER Chip acceleration
[ https://issues.apache.org/jira/browse/HDFS-9529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15049970#comment-15049970 ] wqijun commented on HDFS-9529: -- ok, I will put it in Hadoop Common project > Extend Erasure Code to support POWER Chip acceleration > -- > > Key: HDFS-9529 > URL: https://issues.apache.org/jira/browse/HDFS-9529 > Project: Hadoop HDFS > Issue Type: New Feature > Components: erasure-coding >Affects Versions: 3.0.0 >Reporter: wqijun >Assignee: wqijun > Fix For: 3.0.0 > > > Erasure Code is a very important feature in new HDFS version. This JIRA will > focus on how to extend EC to support multiple types of EC acceleration by C > library and other hardware method, like GPU or FPGA. Compared with > Hadoop-11887, this JIRA will more focus on how to leverage POWER Chip > capability to accelerate the EC calculating. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9519) Some coding improvement in SecondaryNameNode#main
[
https://issues.apache.org/jira/browse/HDFS-9519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiao Chen updated HDFS-9519:
Attachment: HDFS-9519.02.patch
> Some coding improvement in SecondaryNameNode#main
> -
>
> Key: HDFS-9519
> URL: https://issues.apache.org/jira/browse/HDFS-9519
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Yongjun Zhang
>Assignee: Xiao Chen
> Attachments: HDFS-9519.01.patch, HDFS-9519.02.patch
>
>
> Two nits:
> # The checking whether secondary is null is not necessary in the following
> code in SecondaryNameNode.java.
> # The comment in this code seems to imply that "when secondary is not null,
> SNN was stared as a daemon.", and this is not true. Suggest to improve the
> comment to make it clear,
> Assign to Xiao since he worked on HDFS-3059. Thanks Xiao.
> {code}
> if (secondary != null) {
> // The web server is only needed when starting SNN as a daemon,
> // and not needed if called from shell command. Starting the web
> server
> // from shell may fail when getting credentials, if the environment
> // is not set up for it, which is most of the case.
> secondary.startInfoServer();
> secondary.startCheckpointThread();
> secondary.join();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7916) 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for infinite loop
[
https://issues.apache.org/jira/browse/HDFS-7916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050084#comment-15050084
]
Vinayakumar B commented on HDFS-7916:
-
{quote}This patch simply ignores the block when BPServiceActor receives the
RemoteException from namenode.
The namenode got the report once and then it chose not to process that request.
So it doesn't make much sense to add that request again to queue.{quote}
Hi [~yzhangal], This is earlier comment from [~rushabh.shah] about handling
entire RemoteException. Do you find any problem here?
> 'reportBadBlocks' from datanodes to standby Node BPServiceActor goes for
> infinite loop
> --
>
> Key: HDFS-7916
> URL: https://issues.apache.org/jira/browse/HDFS-7916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.7.0
>Reporter: Vinayakumar B
>Assignee: Rushabh S Shah
>Priority: Critical
> Fix For: 2.7.1
>
> Attachments: HDFS-7916-01.patch, HDFS-7916-1.patch
>
>
> if any badblock found, then BPSA for StandbyNode will go for infinite times
> to report it.
> {noformat}2015-03-11 19:43:41,528 WARN
> org.apache.hadoop.hdfs.server.datanode.DataNode: Failed to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> stobdtserver3/10.224.54.70:18010
> org.apache.hadoop.hdfs.server.datanode.BPServiceActorActionException: Failed
> to report bad block
> BP-1384821822-10.224.54.68-1422634566395:blk_1079544278_5812006 to namenode:
> at
> org.apache.hadoop.hdfs.server.datanode.ReportBadBlockAction.reportTo(ReportBadBlockAction.java:63)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.processQueueMessages(BPServiceActor.java:1020)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:762)
> at
> org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:856)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8578) On upgrade, Datanode should process all storage/data dirs in parallel
[
https://issues.apache.org/jira/browse/HDFS-8578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vinayakumar B updated HDFS-8578:
Attachment: HDFS-8578-15.patch
HDFS-8578-branch-2.7-003.patch
Updated patch as suggested by [~szetszwo].
# Introduced config *dfs.datanode.parallel.volumes.load.threads.num* which
defaults to # of data dirs configured.
> On upgrade, Datanode should process all storage/data dirs in parallel
> -
>
> Key: HDFS-8578
> URL: https://issues.apache.org/jira/browse/HDFS-8578
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Reporter: Raju Bairishetti
>Assignee: Vinayakumar B
>Priority: Critical
> Attachments: HDFS-8578-01.patch, HDFS-8578-02.patch,
> HDFS-8578-03.patch, HDFS-8578-04.patch, HDFS-8578-05.patch,
> HDFS-8578-06.patch, HDFS-8578-07.patch, HDFS-8578-08.patch,
> HDFS-8578-09.patch, HDFS-8578-10.patch, HDFS-8578-11.patch,
> HDFS-8578-12.patch, HDFS-8578-13.patch, HDFS-8578-14.patch,
> HDFS-8578-15.patch, HDFS-8578-branch-2.6.0.patch,
> HDFS-8578-branch-2.7-001.patch, HDFS-8578-branch-2.7-002.patch,
> HDFS-8578-branch-2.7-003.patch
>
>
> Right now, during upgrades datanode is processing all the storage dirs
> sequentially. Assume it takes ~20 mins to process a single storage dir then
> datanode which has ~10 disks will take around 3hours to come up.
> *BlockPoolSliceStorage.java*
> {code}
>for (int idx = 0; idx < getNumStorageDirs(); idx++) {
> doTransition(datanode, getStorageDir(idx), nsInfo, startOpt);
> assert getCTime() == nsInfo.getCTime()
> : "Data-node and name-node CTimes must be the same.";
> }
> {code}
> It would save lots of time during major upgrades if datanode process all
> storagedirs/disks parallelly.
> Can we make datanode to process all storage dirs parallelly?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9527) The return type of FSNamesystem.getBlockCollection should be changed to INodeFile
[ https://issues.apache.org/jira/browse/HDFS-9527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050251#comment-15050251 ] Hudson commented on HDFS-9527: -- FAILURE: Integrated in Hadoop-Hdfs-trunk-Java8 #681 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/681/]) HDFS-9527. The return type of FSNamesystem.getBlockCollection should be (szetszwo: rev 132478e805ba0f955345217b8ad87c2d17cccb2d) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DecommissionManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestINodeFile.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestClientProtocolForPipelineRecovery.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Namesystem.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NamenodeFsck.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestReplicationPolicy.java > The return type of FSNamesystem.getBlockCollection should be changed to > INodeFile > - > > Key: HDFS-9527 > URL: https://issues.apache.org/jira/browse/HDFS-9527 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Reporter: Tsz Wo Nicholas Sze >Assignee: Tsz Wo Nicholas Sze >Priority: Minor > Fix For: 2.8.0 > > Attachments: h9527_20151208.patch, h9527_20151209.patch > > > FSNamesystem.getBlockCollection always returns INodeFile. It avoids > unnecessary conversion from BlockCollection to INode/INodeFile after the > change. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-9472) concat() API does not resolve the .reserved path
[
https://issues.apache.org/jira/browse/HDFS-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-9472:
---
Attachment: HDFS-9472-03.patch
> concat() API does not resolve the .reserved path
>
>
> Key: HDFS-9472
> URL: https://issues.apache.org/jira/browse/HDFS-9472
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9472-00.patch, HDFS-9472-01.patch,
> HDFS-9472-02.patch, HDFS-9472-03.patch
>
>
> dfs#concat() API doesn't resolve the {{/.reserved/raw}} path. For example,
> if the input paths of the form {{/.reserved/raw/ezone/a}} then this API
> doesn't work properly. IMHO will discuss here to support this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-9472) concat() API does not resolve the .reserved path
[
https://issues.apache.org/jira/browse/HDFS-9472?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15050261#comment-15050261
]
Uma Maheswara Rao G commented on HDFS-9472:
---
nit:
{code}
throw new IOException("Concat operation doesn't supports "
+ + FSDirectory.DOT_RESERVED_STRING + " relative path : " + target);
{code}
supports -->support
Otherwise +1
> concat() API does not resolve the .reserved path
>
>
> Key: HDFS-9472
> URL: https://issues.apache.org/jira/browse/HDFS-9472
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Rakesh R
>Assignee: Rakesh R
> Attachments: HDFS-9472-00.patch, HDFS-9472-01.patch,
> HDFS-9472-02.patch
>
>
> dfs#concat() API doesn't resolve the {{/.reserved/raw}} path. For example,
> if the input paths of the form {{/.reserved/raw/ezone/a}} then this API
> doesn't work properly. IMHO will discuss here to support this behavior.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
