[jira] [Updated] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8499:
Attachment: HDFS-8499.02.patch
Attaching new patch to address Yi's comment. The {{BlockInfoContiguousFeature}}
class is a helper with static method implementing the shared logic of
{{BIContiguous}} and {{BIContiguousUC}}.
This is basically a multi-inheritance problem. I had an offline discussion with
[~andrew.wang] to compare different options of solving this. One option is
demonstrated in the 02 patch. Another option is to keep the hierarchy in
HDFS-7285 branch, and convert shared UC logic into a feature ([~jingzhao] and I
also discussed this idea in an earlier meetup). I'll post a patch later today
to demonstrate that direction, and we can determine which one is cleaner.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6708) StorageType should be encoded in the block token
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577516#comment-14577516 ] Arpit Agarwal commented on HDFS-6708: - Hi Nicholas, I think so. Since the storage types are not encoded in the block token the client could override the quota checks made by NN. Let me take a look at it for 2.8. StorageType should be encoded in the block token Key: HDFS-6708 URL: https://issues.apache.org/jira/browse/HDFS-6708 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.4.1 Reporter: Arpit Agarwal Assignee: Arpit Agarwal HDFS-6702 is adding support for file creation based on StorageType. The block token is used as a tamper-proof channel for communicating block parameters from the NN to the DN during block creation. The StorageType should be included in this block token. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577601#comment-14577601
]
Xiaoyu Yao commented on HDFS-8551:
--
+1 pending Jenkins.
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577615#comment-14577615
]
Jing Zhao commented on HDFS-8494:
-
Also, we should consider adding a chunk size field to {{StripedBlockProto}} and
removing the cell size field from HdfsFileStatus. In this way we can access the
chunk size information in the storage layer.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8520) Patch for PPC64 block size
[ https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577631#comment-14577631 ] Andrew Wang commented on HDFS-8520: --- Hey Tony, thanks for working on this. I hit Submit Patch since we do need to get a Jenkins run. It'd help also if you based you patch on trunk rather than branch-2.7. The NoMlock manipulator is meant for test environments, and it doesn't actually call mlock. Not related to Windows, just so we don't have to worry about mlock ulimits being set. What issues do you run into when running the tests on PPC64? I'm guessing PageRounder, since it's page size isn't affected by setting the NoMlock manipulator. We should probably combine these two classes, so the PageRounder is also correctly updated when setting the cache manipulator. This would probably let us avoid the test changes. If you want to get serious about not regressing PPC64, seems like we should also add some tests that relate to page size. Having a LargePageCacheManipulator or something would be one way. Patch for PPC64 block size -- Key: HDFS-8520 URL: https://issues.apache.org/jira/browse/HDFS-8520 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Environment: RHEL 7.1 /PPC64 Reporter: Tony Reix Assignee: Tony Reix Fix For: 2.7.1 Attachments: HDFS-8520.patch The attached patch enables Hadoop to work on PPC64. That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64. There are changes in 3 files: - hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE The patch has been built on branch-2.7 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8552) Fix hdfs namenode CLI usage message and its document
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577559#comment-14577559
]
Xiaoyu Yao commented on HDFS-8552:
--
I think we could consolidate the fix together, please update the title as
appropriate.
Fix hdfs namenode CLI usage message and its document
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
2. Many of options are not documented.
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577543#comment-14577543
]
Xiaoyu Yao commented on HDFS-8551:
--
[~brahmareddy], can you fix the checkstyle issue? Otherwise, looks good to me.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:268:
Line is longer than 80 characters (found 83).
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8552) Fix hdfs CLI usage message for namenode and zkfc
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577627#comment-14577627
]
Brahma Reddy Battula commented on HDFS-8552:
[~xyao] Attached patch to address ZKFC and Namenode CLI usage..Y'day only I had
raised for YARN and MR to handle CLI usage message..
Kindly review!!
Fix hdfs CLI usage message for namenode and zkfc
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8552) Fix hdfs namenode CLI usage message and its document
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577546#comment-14577546
]
Xiaoyu Yao commented on HDFS-8552:
--
You are right, I misread the usage strings in the original description and
issue 2 is not a problem. I will update it.
Fix hdfs namenode CLI usage message and its document
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
2. Many of options are not documented.
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8551:
---
Attachment: HDFS-8551-002.patch
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7827) Erasure Coding: support striped blocks in non-protobuf fsimage
[
https://issues.apache.org/jira/browse/HDFS-7827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577605#comment-14577605
]
Jing Zhao commented on HDFS-7827:
-
I just found that for a legacy fsimage we currently always record its
layoutversion as -51 so that we can make sure it cannot be processed by a
protobuf-based image parser. Thus the following code in the parser always
returns false and the parse will fail. I will create a jira to fix this.
{code}
NameNodeLayoutVersion.supports(
NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
{code}
Erasure Coding: support striped blocks in non-protobuf fsimage
--
Key: HDFS-7827
URL: https://issues.apache.org/jira/browse/HDFS-7827
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Hui Zheng
Fix For: HDFS-7285
Attachments: HDFS-7827.000.patch, HDFS-7827.002.patch,
HDFS-7827.003.patch, HDFS-7827.004.patch
HDFS-7749 only adds code to persist striped blocks to protobuf-based fsimage.
We should also add this support to the non-protobuf fsimage since it is still
used for use cases like offline image processing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577558#comment-14577558
]
Zhe Zhang commented on HDFS-8494:
-
We can also consider creating a data structure with both EC schema and
{{cellSize}} to represent _EC policy_ for the file. Something like
{{ErasureCodingZone}}, but minus the {{dir}}. [~vinayrpet] I think we discussed
this under HDFS-8375 and HDFS-8408. What's your take on this?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8552) Fix hdfs namenode CLI usage message and its document
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8552:
-
Description:
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
was:
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented. -
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
Fix hdfs namenode CLI usage message and its document
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577586#comment-14577586
]
Jing Zhao commented on HDFS-8494:
-
Thanks for working on this, [~kaisasak]!
In general I'm not very sure if we need to remove the hard-coded chunk size at
this stage. But if we want to do this, we should record this information in
BlockInfoStriped so that BlockInfoStriped is self explained. I think the basic
idea is that we'd better decouple the logic of the blocks, which belong to the
storage level, from the files. This means if one day we separate the block
management out of the current namenode, the block manager service can provide
enough information for a reader to read the striped block.
For memory footprint, we can easily do some follow-on optimization to save this
extra 4 bytes. E.g., we can store all the existing combinations of (ecschema +
chunk size) somewhere and only record an id in each striped block.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8559) Erasure Coding: fix non-protobuf fsimage for striped blocks
Jing Zhao created HDFS-8559:
---
Summary: Erasure Coding: fix non-protobuf fsimage for striped
blocks
Key: HDFS-8559
URL: https://issues.apache.org/jira/browse/HDFS-8559
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
For a legacy fsimage we currently always record its layoutversion as -51 so
that we can make sure it cannot be processed by a protobuf-based image parser.
Thus the following code in the parser always returns false and the parse will
fail.
{code}
NameNodeLayoutVersion.supports(
NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577674#comment-14577674
]
surendra singh lilhore commented on HDFS-8540:
--
Thanks [~vinayrpet] and [~szetszwo] for reviewing and suggestion.
@[~szetszwo] : you mean it should exit with NO_MOVE_BLOCK when in entire
iteration no blocks are moved.
Mover should exit with NO_MOVE_BLOCK if no block can be moved
-
Key: HDFS-8540
URL: https://issues.apache.org/jira/browse/HDFS-8540
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Reporter: Tsz Wo Nicholas Sze
Assignee: surendra singh lilhore
Attachments: HDFS-8540.patch
When there are files not satisfying their storage policy and no move is
possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
The bug seems in the following code. When StorageTypeDiff is not empty and
scheduleMoves4Block return false, it does not update hasRemaining. Also,
there is no indication of No block can be moved for the entire iteration.
{code}
//Mover.processFile(..)
if (!diff.removeOverlap(true)) {
if (scheduleMoves4Block(diff, lb, ecSchema)) {
hasRemaining |= (diff.existing.size() 1
diff.expected.size() 1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8552) Fix hdfs namenode CLI usage message and its document
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8552:
-
Description:
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented. -
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
was:
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
2. Many of options are not documented.
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] | \n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
Fix hdfs namenode CLI usage message and its document
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented. -
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577607#comment-14577607
]
Zhe Zhang commented on HDFS-8494:
-
Thanks Jing for sharing the thoughts. It's a good idea to store all EC policies
and refer to the IDs. Looks like this can be done in
{{ErasureCodingZoneManager}}, similar to how encryption zones are stored in a
map.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8549) Abort the balancer if an upgrade is in progress
[ https://issues.apache.org/jira/browse/HDFS-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8549: -- Attachment: HDFS-8549.002.patch Fixed a long line and a comment Abort the balancer if an upgrade is in progress --- Key: HDFS-8549 URL: https://issues.apache.org/jira/browse/HDFS-8549 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-8549.001.patch, HDFS-8549.002.patch Running the balancer during an ongoing upgrade has a negative affect, since DNs do not actually delete blocks. This means the balancer is making lots of extra replicas and not actually reducing the disk utilization of over-utilized nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6708) StorageType should be encoded in the block token
[ https://issues.apache.org/jira/browse/HDFS-6708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6708: Target Version/s: 2.8.0 Assignee: Arpit Agarwal StorageType should be encoded in the block token Key: HDFS-6708 URL: https://issues.apache.org/jira/browse/HDFS-6708 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 2.4.1 Reporter: Arpit Agarwal Assignee: Arpit Agarwal HDFS-6702 is adding support for file creation based on StorageType. The block token is used as a tamper-proof channel for communicating block parameters from the NN to the DN during block creation. The StorageType should be included in this block token. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577526#comment-14577526
]
Tsz Wo Nicholas Sze commented on HDFS-8540:
---
The patch computes failToScheduleBlock but failToScheduleBlock is different
from NO_MOVE_BLOCK since we could have both success and fail to schedule
blocks. In this case, it should not return NO_MOVE_BLOCK.
Mover should exit with NO_MOVE_BLOCK if no block can be moved
-
Key: HDFS-8540
URL: https://issues.apache.org/jira/browse/HDFS-8540
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Reporter: Tsz Wo Nicholas Sze
Assignee: surendra singh lilhore
Attachments: HDFS-8540.patch
When there are files not satisfying their storage policy and no move is
possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
The bug seems in the following code. When StorageTypeDiff is not empty and
scheduleMoves4Block return false, it does not update hasRemaining. Also,
there is no indication of No block can be moved for the entire iteration.
{code}
//Mover.processFile(..)
if (!diff.removeOverlap(true)) {
if (scheduleMoves4Block(diff, lb, ecSchema)) {
hasRemaining |= (diff.existing.size() 1
diff.expected.size() 1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8549) Abort the balancer if an upgrade is in progress
[ https://issues.apache.org/jira/browse/HDFS-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8549: -- Status: Patch Available (was: Open) Abort the balancer if an upgrade is in progress --- Key: HDFS-8549 URL: https://issues.apache.org/jira/browse/HDFS-8549 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-8549.001.patch Running the balancer during an ongoing upgrade has a negative affect, since DNs do not actually delete blocks. This means the balancer is making lots of extra replicas and not actually reducing the disk utilization of over-utilized nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8462) Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-8462:
---
Status: Patch Available (was: Open)
Please find the attached patch
Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
---
Key: HDFS-8462
URL: https://issues.apache.org/jira/browse/HDFS-8462
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Akira AJISAKA
Assignee: Jagadesh Kiran N
Attachments: HDFS-8462-00.patch
In Hadoop 2.7.0, WebImageViewer supports the following operations:
* {{GETFILESTATUS}}
* {{LISTSTATUS}}
* {{GETACLSTATUS}}
I'm thinking it would be better for administrators if {{GETXATTRS}} and
{{LISTXATTRS}} are supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577593#comment-14577593
]
Brahma Reddy Battula commented on HDFS-8551:
[~xyao] Attached the patch to address checkstyle..Kindly review..Thanks!!
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8520) Patch for PPC64 block size
[ https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8520: -- Status: Patch Available (was: Open) Patch for PPC64 block size -- Key: HDFS-8520 URL: https://issues.apache.org/jira/browse/HDFS-8520 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Environment: RHEL 7.1 /PPC64 Reporter: Tony Reix Assignee: Tony Reix Fix For: 2.7.1 Attachments: HDFS-8520.patch The attached patch enables Hadoop to work on PPC64. That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64. There are changes in 3 files: - hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE The patch has been built on branch-2.7 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-8520) Patch for PPC64 block size
[ https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-8520: - Assignee: Tony Reix Patch for PPC64 block size -- Key: HDFS-8520 URL: https://issues.apache.org/jira/browse/HDFS-8520 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.7.1 Environment: RHEL 7.1 /PPC64 Reporter: Tony Reix Assignee: Tony Reix Fix For: 2.7.1 Attachments: HDFS-8520.patch The attached patch enables Hadoop to work on PPC64. That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64. There are changes in 3 files: - hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java - hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE The patch has been built on branch-2.7 . -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8552) Fix hdfs namenode CLI usage message and its document
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8552:
---
Attachment: HDFS-8552-002.patch
Fix hdfs namenode CLI usage message and its document
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8552) Fix hdfs CLI usage message for namenode and zkfc
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8552:
---
Summary: Fix hdfs CLI usage message for namenode and zkfc (was: Fix hdfs
namenode CLI usage message and its document)
Fix hdfs CLI usage message for namenode and zkfc
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577545#comment-14577545
]
Zhe Zhang commented on HDFS-8494:
-
Thanks for the discussion folks.
I remember discussing this in an offline meetup a while ago with [~jingzhao]
and [~szetszwo]. The conclusion at that time was that we should avoid storing
EC schema for each block.
But I think the question is worth discussing again. A medium size cluster could
have several 100M blocks. 4b a block translates to a few GB of memory overhead.
On the other side of the tradeoff, as Walter analyzed, we need to go through a
few hops to lookup the {{cellSize}}. The most expensive part is probably the
path traversal and {{getECZone}}. So if we don't need to get {{cellSize}}
frequently, I'd suggest keeping in in the zone. We can certainly add a public
method in {{BlockInfoStriped}} to simplify the code.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8546) Use try with resources in DataStorage and Storage
[ https://issues.apache.org/jira/browse/HDFS-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-8546: -- Attachment: HDFS-8546.004.patch TY for the review again, I removed the extra finally. OverlappingFileLockException is a JDK class that doesn't have a constructor that takes a String, so could not address that one. Fixed the checkstyle issues too. Use try with resources in DataStorage and Storage - Key: HDFS-8546 URL: https://issues.apache.org/jira/browse/HDFS-8546 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Priority: Minor Attachments: HDFS-8546.001.patch, HDFS-8546.002.patch, HDFS-8546.003.patch, HDFS-8546.004.patch We have some old-style try/finally to close files in DataStorage and Storage, let's update them. Also a few small cleanups: * Actually check that tryLock returns a FileLock in isPreUpgradableLayout * Remove unused parameter from writeProperties * Add braces for one-line if statements per coding style -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8462) Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jagadesh Kiran N updated HDFS-8462:
---
Attachment: HDFS-8462-00.patch
I have attached the patch please review
Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
---
Key: HDFS-8462
URL: https://issues.apache.org/jira/browse/HDFS-8462
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Akira AJISAKA
Assignee: Jagadesh Kiran N
Attachments: HDFS-8462-00.patch
In Hadoop 2.7.0, WebImageViewer supports the following operations:
* {{GETFILESTATUS}}
* {{LISTSTATUS}}
* {{GETACLSTATUS}}
I'm thinking it would be better for administrators if {{GETXATTRS}} and
{{LISTXATTRS}} are supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577688#comment-14577688
]
Tsz Wo Nicholas Sze commented on HDFS-8540:
---
Yes, NO_MOVE_BLOCK means that No block can be moved. If some blocks can be
moved in current iteration, space may be freed up so that the blocks failed to
move may possibly be moved in the next iteration.
Mover should exit with NO_MOVE_BLOCK if no block can be moved
-
Key: HDFS-8540
URL: https://issues.apache.org/jira/browse/HDFS-8540
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Reporter: Tsz Wo Nicholas Sze
Assignee: surendra singh lilhore
Attachments: HDFS-8540.patch
When there are files not satisfying their storage policy and no move is
possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
The bug seems in the following code. When StorageTypeDiff is not empty and
scheduleMoves4Block return false, it does not update hasRemaining. Also,
there is no indication of No block can be moved for the entire iteration.
{code}
//Mover.processFile(..)
if (!diff.removeOverlap(true)) {
if (scheduleMoves4Block(diff, lb, ecSchema)) {
hasRemaining |= (diff.existing.size() 1
diff.expected.size() 1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8549) Abort the balancer if an upgrade is in progress
[
https://issues.apache.org/jira/browse/HDFS-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577909#comment-14577909
]
Hadoop QA commented on HDFS-8549:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 49s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 1s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 21s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 17s | The applied patch generated 1
new checkstyle issues (total was 54, now 55). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 22s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 161m 55s | Tests failed in hadoop-hdfs. |
| | | 209m 17s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738103/HDFS-8549.001.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11274/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11274/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11274/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11274/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11274/console |
This message was automatically generated.
Abort the balancer if an upgrade is in progress
---
Key: HDFS-8549
URL: https://issues.apache.org/jira/browse/HDFS-8549
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.7.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-8549.001.patch, HDFS-8549.002.patch
Running the balancer during an ongoing upgrade has a negative affect, since
DNs do not actually delete blocks. This means the balancer is making lots of
extra replicas and not actually reducing the disk utilization of
over-utilized nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7923: --- Attachment: HDFS-7923.007.patch The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages --- Key: HDFS-7923 URL: https://issues.apache.org/jira/browse/HDFS-7923 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, HDFS-7923.006.patch, HDFS-7923.007.patch The DataNodes should rate-limit their full block reports. They can do this by first sending a heartbeat message to the NN with an optional boolean set which requests permission to send a full block report. If the NN responds with another optional boolean set, the DN will send an FBR... if not, it will wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8549) Abort the balancer if an upgrade is in progress
[ https://issues.apache.org/jira/browse/HDFS-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578019#comment-14578019 ] Colin Patrick McCabe commented on HDFS-8549: So I guess the idea here is that running the balancer while we still have the previous directory around doesn't actually make sense most of the time. When the balancer tries to move a block, it copies it to the new datanode and then deletes it from the current/ directory of the old datanode. But when the previous directory is around, it won't really be deleted from the source node. So the space consumption will not decrease on the source node. Arguably, running the balancer might still serve a useful function in bringing data to the nodes where it needs to be to be more balanced. When we finish the upgrade, we can then delete all the duplicates when we delete the previous directories. So, I think your change makes sense as the default behavior, but I think we might want a flag to provide a manual override. Can you provide a force flag so that we can run the balancer even during an upgrade? +1 once that's added Abort the balancer if an upgrade is in progress --- Key: HDFS-8549 URL: https://issues.apache.org/jira/browse/HDFS-8549 Project: Hadoop HDFS Issue Type: Improvement Components: balancer mover Affects Versions: 2.7.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: HDFS-8549.001.patch, HDFS-8549.002.patch Running the balancer during an ongoing upgrade has a negative affect, since DNs do not actually delete blocks. This means the balancer is making lots of extra replicas and not actually reducing the disk utilization of over-utilized nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577986#comment-14577986 ] Colin Patrick McCabe commented on HDFS-7923: New patch fixes unit test failures. TestBpServiceActorScheduler needed to be reverted to the old version (since the BR timing change was also removed from the latest version), TestDatanodeManager needed to be tweaked because of the way it uses mocks, to avoid a null pointer exception. There was also a bug where we'd get an NPE when removing elements from the internal linked lists in the BlockReportLeaseManager which was causing unit test failures. The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages --- Key: HDFS-7923 URL: https://issues.apache.org/jira/browse/HDFS-7923 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.8.0 Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch, HDFS-7923.006.patch The DataNodes should rate-limit their full block reports. They can do this by first sending a heartbeat message to the NN with an optional boolean set which requests permission to send a full block report. If the NN responds with another optional boolean set, the DN will send an FBR... if not, it will wait until later. This can be done compatibly with optional fields. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8462) Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
[
https://issues.apache.org/jira/browse/HDFS-8462?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577940#comment-14577940
]
Hadoop QA commented on HDFS-8462:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 25s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 47s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 52s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated
15 new checkstyle issues (total was 136, now 150). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 19s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 22s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 53s | Tests passed in hadoop-hdfs.
|
| | | 210m 29s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738401/HDFS-8462-00.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11275/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11275/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11275/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11275/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11275/console |
This message was automatically generated.
Implement GETXATTRS and LISTXATTRS operation for WebImageViewer
---
Key: HDFS-8462
URL: https://issues.apache.org/jira/browse/HDFS-8462
Project: Hadoop HDFS
Issue Type: New Feature
Reporter: Akira AJISAKA
Assignee: Jagadesh Kiran N
Attachments: HDFS-8462-00.patch
In Hadoop 2.7.0, WebImageViewer supports the following operations:
* {{GETFILESTATUS}}
* {{LISTSTATUS}}
* {{GETACLSTATUS}}
I'm thinking it would be better for administrators if {{GETXATTRS}} and
{{LISTXATTRS}} are supported.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8480) Fix performance and timeout issues in HDFS-7929: use hard-links instead of copying edit logs
[
https://issues.apache.org/jira/browse/HDFS-8480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577993#comment-14577993
]
Colin Patrick McCabe commented on HDFS-8480:
+1, pending adding the unit test you talked about earlier. Thanks, [~zhz].
Fix performance and timeout issues in HDFS-7929: use hard-links instead of
copying edit logs
Key: HDFS-8480
URL: https://issues.apache.org/jira/browse/HDFS-8480
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Critical
Attachments: HDFS-8480.00.patch
HDFS-7929 copies existing edit logs to the storage directory of the upgraded
{{NameNode}}. This slows down the upgrade process. This JIRA aims to use
hard-linking instead of per-op copying to achieve the same goal.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8520) Patch for PPC64 block size
[
https://issues.apache.org/jira/browse/HDFS-8520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577976#comment-14577976
]
Hadoop QA commented on HDFS-8520:
-
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 32s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 3s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 22m 46s | Tests passed in
hadoop-common. |
| {color:green}+1{color} | hdfs tests | 161m 39s | Tests passed in hadoop-hdfs.
|
| | | 231m 59s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12736863/HDFS-8520.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11276/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11276/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11276/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11276/console |
This message was automatically generated.
Patch for PPC64 block size
--
Key: HDFS-8520
URL: https://issues.apache.org/jira/browse/HDFS-8520
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.7.1
Environment: RHEL 7.1 /PPC64
Reporter: Tony Reix
Assignee: Tony Reix
Fix For: 2.7.1
Attachments: HDFS-8520.patch
The attached patch enables Hadoop to work on PPC64.
That deals with SystemPageSize and BloclSize , which are not 4096 on PPC64.
There are changes in 3 files:
-
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/io/nativeio/NativeIO.java
-
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/datanode/TestFsDatasetCache.java
-
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCacheDirectives.java
where 4096 is replaced by getOperatingSystemPageSize() or by using PAGE_SIZE
The patch has been built on branch-2.7 .
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8552) Fix hdfs CLI usage message for namenode and zkfc
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578002#comment-14578002
]
Hadoop QA commented on HDFS-8552:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 9s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 26s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 37s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 3m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 32s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 5m 4s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | common tests | 23m 44s | Tests passed in
hadoop-common. |
| {color:red}-1{color} | hdfs tests | 161m 7s | Tests failed in hadoop-hdfs. |
| | | 231m 56s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestClusterId |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738410/HDFS-8552-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| hadoop-common test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11277/artifact/patchprocess/testrun_hadoop-common.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11277/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11277/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf906.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11277/console |
This message was automatically generated.
Fix hdfs CLI usage message for namenode and zkfc
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade
[jira] [Updated] (HDFS-8559) Erasure Coding: fix non-protobuf fsimage for striped blocks
[
https://issues.apache.org/jira/browse/HDFS-8559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-8559:
Attachment: HDFS-8559.000.patch
For now we can save and load striped blocks just as contiguous blocks in a
legacy fsimage. Since the legacy fsimage currently is only used for offline
analysis, not exposing striped blocks information there should be fine.
Erasure Coding: fix non-protobuf fsimage for striped blocks
---
Key: HDFS-8559
URL: https://issues.apache.org/jira/browse/HDFS-8559
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-8559.000.patch
For a legacy fsimage we currently always record its layoutversion as -51 so
that we can make sure it cannot be processed by a protobuf-based image
parser. Thus the following code in the parser always returns false and the
parse will fail.
{code}
NameNodeLayoutVersion.supports(
NameNodeLayoutVersion.Feature.ERASURE_CODING, imgVersion)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577905#comment-14577905
]
Hadoop QA commented on HDFS-8551:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 0s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 39s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 56s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 2
new checkstyle issues (total was 142, now 143). |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 18s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 161m 22s | Tests failed in hadoop-hdfs. |
| | | 208m 22s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738403/HDFS-8551-002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11273/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11273/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11273/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11273/console |
This message was automatically generated.
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8546) Use try with resources in DataStorage and Storage
[
https://issues.apache.org/jira/browse/HDFS-8546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577904#comment-14577904
]
Hadoop QA commented on HDFS-8546:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 45s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 29s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 38s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 18s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 20s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 161m 46s | Tests passed in hadoop-hdfs.
|
| | | 208m 4s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738402/HDFS-8546.004.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11272/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11272/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11272/console |
This message was automatically generated.
Use try with resources in DataStorage and Storage
-
Key: HDFS-8546
URL: https://issues.apache.org/jira/browse/HDFS-8546
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 2.7.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Priority: Minor
Attachments: HDFS-8546.001.patch, HDFS-8546.002.patch,
HDFS-8546.003.patch, HDFS-8546.004.patch
We have some old-style try/finally to close files in DataStorage and Storage,
let's update them.
Also a few small cleanups:
* Actually check that tryLock returns a FileLock in isPreUpgradableLayout
* Remove unused parameter from writeProperties
* Add braces for one-line if statements per coding style
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8549) Abort the balancer if an upgrade is in progress
[
https://issues.apache.org/jira/browse/HDFS-8549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14577964#comment-14577964
]
Hadoop QA commented on HDFS-8549:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 39s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 34s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 2m 11s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 35s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 32s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 15s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 13s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 160m 54s | Tests failed in hadoop-hdfs. |
| | | 206m 53s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738412/HDFS-8549.002.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11278/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11278/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11278/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf902.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11278/console |
This message was automatically generated.
Abort the balancer if an upgrade is in progress
---
Key: HDFS-8549
URL: https://issues.apache.org/jira/browse/HDFS-8549
Project: Hadoop HDFS
Issue Type: Improvement
Components: balancer mover
Affects Versions: 2.7.0
Reporter: Andrew Wang
Assignee: Andrew Wang
Attachments: HDFS-8549.001.patch, HDFS-8549.002.patch
Running the balancer during an ongoing upgrade has a negative affect, since
DNs do not actually delete blocks. This means the balancer is making lots of
extra replicas and not actually reducing the disk utilization of
over-utilized nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Moved] (HDFS-8560) Document that DN max locked memory must be configured to use RAM disk
[ https://issues.apache.org/jira/browse/HDFS-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal moved HADOOP-12075 to HDFS-8560: -- Component/s: (was: documentation) documentation Target Version/s: 2.8.0 (was: 2.8.0) Affects Version/s: (was: 2.8.0) 2.8.0 Key: HDFS-8560 (was: HADOOP-12075) Project: Hadoop HDFS (was: Hadoop Common) Document that DN max locked memory must be configured to use RAM disk - Key: HDFS-8560 URL: https://issues.apache.org/jira/browse/HDFS-8560 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.8.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal HDFS-6599 introduced the requirement that max locked memory must be configured to use RAM disk storage via the LAZY_PERSIST storage policy. We need to document it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-8561) checkNNStartup should throw a RetriableException
Arpit Agarwal created HDFS-8561:
---
Summary: checkNNStartup should throw a RetriableException
Key: HDFS-8561
URL: https://issues.apache.org/jira/browse/HDFS-8561
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Priority: Critical
{{checkNNStartup}} currently throws an {{IOException}} if services have not yet
been started.
This propagates as a RemoteException to the client which is not retriable with
the default retry policy in RetryUtils.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8553) Document hdfs class path options
[
https://issues.apache.org/jira/browse/HDFS-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578041#comment-14578041
]
Chris Nauroth commented on HDFS-8553:
-
I have committed this to trunk and branch-2.
Document hdfs class path options
Key: HDFS-8553
URL: https://issues.apache.org/jira/browse/HDFS-8553
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8553-002.patch, HDFS-8553.patch
--global, --jar options are not documented.
{code}
$ hdfs classpath --help
classpath [--glob|--jar path|-h|--help] :
Prints the classpath needed to get the Hadoop jar and the required
libraries.
Options:
--glob expand wildcards
--jar path write classpath as manifest in jar named path
-h, --help print help
{code}
current document:
{code}
User Commands
Commands useful for users of a hadoop cluster.
classpath
Usage: hdfs classpath
Prints the class path needed to get the Hadoop jar and the required libraries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8560) Document that DN max locked memory must be configured to use RAM disk
[ https://issues.apache.org/jira/browse/HDFS-8560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-8560: Description: HDFS-6919 introduced the requirement that max locked memory must be configured to use RAM disk storage via the LAZY_PERSIST storage policy. We need to document it. was: HDFS-6599 introduced the requirement that max locked memory must be configured to use RAM disk storage via the LAZY_PERSIST storage policy. We need to document it. Document that DN max locked memory must be configured to use RAM disk - Key: HDFS-8560 URL: https://issues.apache.org/jira/browse/HDFS-8560 Project: Hadoop HDFS Issue Type: Bug Components: documentation Affects Versions: 2.8.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal HDFS-6919 introduced the requirement that max locked memory must be configured to use RAM disk storage via the LAZY_PERSIST storage policy. We need to document it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8246) Get HDFS file name based on block pool id and block id
[ https://issues.apache.org/jira/browse/HDFS-8246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578122#comment-14578122 ] Colin Patrick McCabe commented on HDFS-8246: I think this API should be accessible to everyone, not just admins. It can help eliminate race conditions. Let's say you have some daemon that runs with superuser permissions that is trying to verify that some user U has permission to access /a/b. Then you have an access pattern like this: 1. daemon checks /a to ensure U has permissions to access it 2. daemon checks /a/b A malicious user might be able to insert a step 1a. 1a. malicious user moves /a to /a2 and moves /something-else to /a Then the malicious user would get access to /something-else/b even if he doesn't have access to /something-else. There are a lot of scenarios you can set up like this. Any time you have a check of a path, followed by a use of that path, there can be a race condition if you are not refering to inodes by number. bq. What should it return if a block belong to multiple files (current file and snapshotted files, or hard linked files when we support hard link in the future)? Then I think it should return multiple paths, including some to snapshot directories. Get HDFS file name based on block pool id and block id -- Key: HDFS-8246 URL: https://issues.apache.org/jira/browse/HDFS-8246 Project: Hadoop HDFS Issue Type: New Feature Components: HDFS, hdfs-client, namenode Reporter: feng xu Assignee: feng xu Labels: BB2015-05-TBR Attachments: HDFS-8246.0.patch This feature provides HDFS shell command and C/Java API to retrieve HDFS file name based on block pool id and block id. 1. The Java API in class DistributedFileSystem public String getFileName(String poolId, long blockId) throws IOException 2. The C API in hdfs.c char* hdfsGetFileName(hdfsFS fs, const char* poolId, int64_t blockId) 3. The HDFS shell command hdfs dfs [generic options] -fn poolId blockId This feature is useful if you have HDFS block file name in local file system and want to find out the related HDFS file name in HDFS name space (http://stackoverflow.com/questions/10881449/how-to-find-file-from-blockname-in-hdfs-hadoop). Each HDFS block file name in local file system contains both block pool id and block id, for sample HDFS block file name /hdfs/1/hadoop/hdfs/data/current/BP-97622798-10.3.11.84-1428081035160/current/finalized/subdir0/subdir0/blk_1073741825, the block pool id is BP-97622798-10.3.11.84-1428081035160 and the block id is 1073741825. The block pool id is uniquely related to a HDFS name node/name space, and the block id is uniquely related to a HDFS file within a HDFS name node/name space, so the combination of block pool id and a block id is uniquely related a HDFS file name. The shell command and C/Java API do not map the block pool id to name node, so it’s user’s responsibility to talk to the correct name node in federation environment that has multiple name nodes. The block pool id is used by name node to check if the user is talking with the correct name node. The implementation is straightforward. The client request to get HDFS file name reaches the new method String getFileName(String poolId, long blockId) in FSNamesystem in name node through RPC, and the new method does the followings, (1) Validate the block pool id. (2) Create Block based on the block id. (3) Get BlockInfoContiguous from Block. (4) Get BlockCollection from BlockInfoContiguous. (5) Get file name from BlockCollection. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8543) Erasure Coding: processOverReplicatedBlock() handles striped block
[ https://issues.apache.org/jira/browse/HDFS-8543?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578160#comment-14578160 ] Walter Su commented on HDFS-8543: - The problem of the old code: Assume blockGroup=\[0, 1, 2, 3\]. We lose idx=2, and are left \[0, 1, 3\]. After recovery, we get\[0, 1, 2, 3\]. Assume dead DN comes back. We have\[0, 1, 2, 3, 2\]. The old processOverReplicatedBlock() may delete idx=1, so we are left \[0, 2, 3, 2\], we have to recover it again. It could be an endless loop. Erasure Coding: processOverReplicatedBlock() handles striped block -- Key: HDFS-8543 URL: https://issues.apache.org/jira/browse/HDFS-8543 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Walter Su Assignee: Walter Su Attachments: HDFS-8543-HDFS-7285.01.patch striped block group could be over replicated when: 1.dead DN comes back. 2. Balancer/Mover copies block before deletes it. This jira add logic for processOverReplicatedBlock() handling striped block -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8515) Abstract a DTP/2 HTTP/2 server
[ https://issues.apache.org/jira/browse/HDFS-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HDFS-8515: Attachment: HDFS-8515.patch A patch for review. Will add more testcases later. Abstract a DTP/2 HTTP/2 server -- Key: HDFS-8515 URL: https://issues.apache.org/jira/browse/HDFS-8515 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HDFS-8515.patch Discussed in HDFS-8471. https://issues.apache.org/jira/browse/HDFS-8471?focusedCommentId=14568196page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568196 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8561) checkNNStartup should throw a RetriableException
[
https://issues.apache.org/jira/browse/HDFS-8561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578199#comment-14578199
]
Arpit Agarwal commented on HDFS-8561:
-
Effectively a regression since 2.6, although 2.6 was broken differently.
One way to fix it is have checkNNState throw RetriableException and also fix
the default retry policy.
checkNNStartup should throw a RetriableException
Key: HDFS-8561
URL: https://issues.apache.org/jira/browse/HDFS-8561
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.7.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
Priority: Critical
{{checkNNStartup}} currently throws an {{IOException}} if services have not
yet been started.
This propagates as a RemoteException to the client which is not retriable
with the default retry policy in RetryUtils.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8499:
Attachment: HDFS-8499.03.patch
Uploading new patch to address test failures. They were caused by a wrong
assert and cast.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
[
https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578204#comment-14578204
]
Hadoop QA commented on HDFS-7923:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 43s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 16 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 41s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 11s | The applied patch generated
23 new checkstyle issues (total was 1341, now 1355). |
| {color:red}-1{color} | whitespace | 0m 8s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 16s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 160m 55s | Tests passed in hadoop-hdfs.
|
| | | 207m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738456/HDFS-7923.007.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 84ba1a7 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11280/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11280/artifact/patchprocess/whitespace.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11280/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11280/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11280/console |
This message was automatically generated.
The DataNodes should rate-limit their full block reports by asking the NN on
heartbeat messages
---
Key: HDFS-7923
URL: https://issues.apache.org/jira/browse/HDFS-7923
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: 2.8.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch,
HDFS-7923.002.patch, HDFS-7923.003.patch, HDFS-7923.004.patch,
HDFS-7923.006.patch, HDFS-7923.007.patch
The DataNodes should rate-limit their full block reports. They can do this
by first sending a heartbeat message to the NN with an optional boolean set
which requests permission to send a full block report. If the NN responds
with another optional boolean set, the DN will send an FBR... if not, it will
wait until later. This can be done compatibly with optional fields.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578100#comment-14578100
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 27s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:red}-1{color} | javac | 1m 31s | The patch appears to cause the
build to fail. |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738465/HDFS-8499.UCFeature.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 025a3a8 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11281/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578110#comment-14578110
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 14s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 51s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 20s | The applied patch generated
30 new checkstyle issues (total was 1136, now 1143). |
| {color:green}+1{color} | whitespace | 0m 5s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 37s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 17s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 19s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 208m 4s | Tests failed in hadoop-hdfs. |
| | | 255m 31s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.hdfs.server.namenode.TestFileTruncate |
| | hadoop.hdfs.TestReadWhileWriting |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestRbwSpaceReservation |
| | hadoop.hdfs.server.namenode.ha.TestQuotasWithHA |
| | hadoop.hdfs.server.blockmanagement.TestBlockTokenWithDFS |
| | hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes |
| | hadoop.hdfs.TestClientProtocolForPipelineRecovery |
| | hadoop.hdfs.TestFileAppend |
| | hadoop.hdfs.TestEncryptedTransfer |
| | hadoop.hdfs.security.TestDelegationTokenForProxyUser |
| | hadoop.hdfs.web.TestWebHdfsFileSystemContract |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA |
| | hadoop.hdfs.TestBlockReaderLocalLegacy |
| | hadoop.hdfs.TestDFSUpgradeFromImage |
| | hadoop.hdfs.TestPersistBlocks |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.server.namenode.ha.TestPipelinesFailover |
| | hadoop.hdfs.TestHDFSFileSystemContract |
| | hadoop.hdfs.server.balancer.TestBalancer |
| | hadoop.hdfs.TestEncryptionZones |
| | hadoop.hdfs.TestAppendDifferentChecksum |
| | hadoop.hdfs.server.blockmanagement.TestReplicationPolicy |
| | hadoop.hdfs.TestFileAppend2 |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.TestFileAppend3 |
| |
hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
|
| | hadoop.hdfs.server.namenode.TestSnapshotPathINodes |
| | hadoop.hdfs.TestGetFileChecksum |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.fs.permission.TestStickyBit |
| | hadoop.hdfs.server.namenode.ha.TestHASafeMode |
| | hadoop.hdfs.server.namenode.TestDiskspaceQuotaUpdate |
| | hadoop.hdfs.server.namenode.TestFSImageWithSnapshot |
| | hadoop.hdfs.TestDataTransferProtocol |
| | hadoop.hdfs.TestDFSShell |
| | hadoop.hdfs.TestPipelines |
| | hadoop.fs.TestHDFSFileContextMainOperations |
| | hadoop.hdfs.server.namenode.ha.TestHAAppend |
| | hadoop.cli.TestHDFSCLI |
| | hadoop.hdfs.TestReplaceDatanodeOnFailure |
| | hadoop.hdfs.server.namenode.TestAddBlock |
| | hadoop.hdfs.server.namenode.ha.TestDNFencing |
| | hadoop.hdfs.TestLeaseRecovery |
| Timed out tests | org.apache.hadoop.hdfs.TestLeaseRecovery2 |
| | org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer |
| | org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738413/HDFS-8499.02.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0e80d51 |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11279/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux
[jira] [Updated] (HDFS-8553) Document hdfs class path options
[
https://issues.apache.org/jira/browse/HDFS-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-8553:
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
+1 for the patch. Brahma, thank you for the contribution. Xiaoyu, thank you
for code reviewing.
Document hdfs class path options
Key: HDFS-8553
URL: https://issues.apache.org/jira/browse/HDFS-8553
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8553-002.patch, HDFS-8553.patch
--global, --jar options are not documented.
{code}
$ hdfs classpath --help
classpath [--glob|--jar path|-h|--help] :
Prints the classpath needed to get the Hadoop jar and the required
libraries.
Options:
--glob expand wildcards
--jar path write classpath as manifest in jar named path
-h, --help print help
{code}
current document:
{code}
User Commands
Commands useful for users of a hadoop cluster.
classpath
Usage: hdfs classpath
Prints the class path needed to get the Hadoop jar and the required libraries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8553) Document hdfs class path options
[
https://issues.apache.org/jira/browse/HDFS-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-8553:
Component/s: documentation
Document hdfs class path options
Key: HDFS-8553
URL: https://issues.apache.org/jira/browse/HDFS-8553
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8553-002.patch, HDFS-8553.patch
--global, --jar options are not documented.
{code}
$ hdfs classpath --help
classpath [--glob|--jar path|-h|--help] :
Prints the classpath needed to get the Hadoop jar and the required
libraries.
Options:
--glob expand wildcards
--jar path write classpath as manifest in jar named path
-h, --help print help
{code}
current document:
{code}
User Commands
Commands useful for users of a hadoop cluster.
classpath
Usage: hdfs classpath
Prints the class path needed to get the Hadoop jar and the required libraries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8553) Document hdfs class path options
[
https://issues.apache.org/jira/browse/HDFS-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578044#comment-14578044
]
Hudson commented on HDFS-8553:
--
FAILURE: Integrated in Hadoop-trunk-Commit #7992 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7992/])
HDFS-8553. Document hdfs class path options. Contributed by Brahma Reddy
Battula. (cnauroth: rev d2832b3d4243c6c470c774bc33fd13f70b3e7b72)
* hadoop-hdfs-project/hadoop-hdfs/src/site/markdown/HDFSCommands.md
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Document hdfs class path options
Key: HDFS-8553
URL: https://issues.apache.org/jira/browse/HDFS-8553
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8553-002.patch, HDFS-8553.patch
--global, --jar options are not documented.
{code}
$ hdfs classpath --help
classpath [--glob|--jar path|-h|--help] :
Prints the classpath needed to get the Hadoop jar and the required
libraries.
Options:
--glob expand wildcards
--jar path write classpath as manifest in jar named path
-h, --help print help
{code}
current document:
{code}
User Commands
Commands useful for users of a hadoop cluster.
classpath
Usage: hdfs classpath
Prints the class path needed to get the Hadoop jar and the required libraries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8499:
Attachment: HDFS-8499.UCFeature.patch
I'm halfway through implementing the 2nd option discussed above, where
{{BlockInfoUC}} is an interface and shared UC logic is moved to a feature
(attaching the {{UCFeature}} patch). It is already messier than the 1st option
(02 patch).
Here's a basic comparison of the 2 options to solve the multi-inheritance
problem:
# UC through inheritance, Striped/Contiguous logic a feature/helper:
{code}
BlockInfo (abstract)
/ | \
BlockInfoStriped |BlockInfoContiguous
|
BlockInfoUC(abstract)
/ \
BlockInfoStripedUC BlockInfoContiguousUC
{code}
#* (y) Existing trunk codes using the abstractions of {{BlockInfo}} and
{{BlockInfoUC}} remain unchanged.
#* (y) The static methods in Striped/Contiguous helper classes are relatively
simple
#* (n) Weakening access permission for internal states, especially {{triplets}}
to package level. We could create a package just for {{BlockInfo}} related
classes but that's more changes.
#* (n) Losing the abstraction of _striped complete or UC blocks_. The
helper/feature class for striped blocks needs to maintain {{indices}}.
# Striped/Contiguous through inheritance, UC as a feature:
{code}
BlockInfo (abstract)
/ \
BlockInfoStriped BlockInfoContiguous
||
| BlockInfoUC |
| (interface) |
| / \ |
BlockInfoStripedUC BlockInfoContiguousUC
{code}
(y) Closer to current HDFS-7285 code
(y) Preserving the shared {{indices}} field in {{BlockInfoStriped}} and
{{BlockInfoStripedUC}}.
(n) There is more shared UC logic than Striped/Contiguous logic. The
{{BlockInfoUCFeature}} class itself is more complex than
{{BlockInfoContiguous/StripedFeature}}.
(n) Involves messy change to convert between {{BlockInfoUC}} and
{{BlockInfo}}. For example, method signatures, like
{{FSNameSystem#isInSnapshot}}, need to be changed because {{BlockInfoUC}} is no
longer a subclass of {{BlockInfo}}.
Looks to me option #1 will probably minimize changes to trunk and make
reviewing easier. Would like to hear more opinions. Thanks!
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8515) Abstract a DTP/2 HTTP/2 server
[ https://issues.apache.org/jira/browse/HDFS-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang updated HDFS-8515: Status: Patch Available (was: Open) Abstract a DTP/2 HTTP/2 server -- Key: HDFS-8515 URL: https://issues.apache.org/jira/browse/HDFS-8515 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Duo Zhang Assignee: Duo Zhang Attachments: HDFS-8515.patch Discussed in HDFS-8471. https://issues.apache.org/jira/browse/HDFS-8471?focusedCommentId=14568196page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568196 -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7966) New Data Transfer Protocol via HTTP/2
[ https://issues.apache.org/jira/browse/HDFS-7966?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578303#comment-14578303 ] Haohui Mai commented on HDFS-7966: -- Sorry for the late reply. Get caught up by multiple things. bq. 2 - If you're using your own payload encoding then tagging flush points in a streaming RPC seems pretty trivial. Thanks very much for the information. Based on the information it looks like that it is possible to use a standard GRPC client to talk the new DTP protocol. It would save a lot of effort on implementing a client of the DTP protocol. I really appreciate if there are any pointers to the client / server code. bq. Generally interested in progress if any. No harm if none. Thanks. Currently [~Apache9] is making progress on HDFS-8515 and HDFS-8471. I have been closely working with [~Apache9] on this. Hopefully we can get things committed soon and continue to make progress. New Data Transfer Protocol via HTTP/2 - Key: HDFS-7966 URL: https://issues.apache.org/jira/browse/HDFS-7966 Project: Hadoop HDFS Issue Type: New Feature Reporter: Haohui Mai Assignee: Qianqian Shi Labels: gsoc, gsoc2015, mentor Attachments: GSoC2015_Proposal.pdf The current Data Transfer Protocol (DTP) implements a rich set of features that span across multiple layers, including: * Connection pooling and authentication (session layer) * Encryption (presentation layer) * Data writing pipeline (application layer) All these features are HDFS-specific and defined by implementation. As a result it requires non-trivial amount of work to implement HDFS clients and servers. This jira explores to delegate the responsibilities of the session and presentation layers to the HTTP/2 protocol. Particularly, HTTP/2 handles connection multiplexing, QoS, authentication and encryption, reducing the scope of DTP to the application layer only. By leveraging the existing HTTP/2 library, it should simplify the implementation of both HDFS clients and servers. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-8499:
Attachment: HDFS-8499.04.patch
Uploading 04 patch with a few polishments:
# Fix findbug issues
# Add Javadocs to new classes
# Per Yi's suggestion, searched for all text occurrences of
{{BlockInfoContiguousUnderConstruction}} (there's only one)
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.04.patch,
HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7285: Attachment: HDFSErasureCodingPhaseITestPlan.pdf Uploading a test plan for phase I of the feature (thanks [~drankye] for filling in the details on codec testing). Any questions and comments are very welcome. When we move on to follow-on optimizations (HDFS-8031) and phase II of the erasure coding feature (HDFS-8030) I will post additional test plans. Erasure Coding Support inside HDFS -- Key: HDFS-7285 URL: https://issues.apache.org/jira/browse/HDFS-7285 Project: Hadoop HDFS Issue Type: New Feature Reporter: Weihua Jiang Assignee: Zhe Zhang Attachments: ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch, HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf, HDFSErasureCodingPhaseITestPlan.pdf, fsimage-analysis-20150105.pdf Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data. Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back. We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8552) Fix hdfs CLI usage message for namenode and zkfc
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578352#comment-14578352
]
Hudson commented on HDFS-8552:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #7995 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/7995/])
HDFS-8552. Fix hdfs CLI usage message for namenode and zkfc. Contributed by
Brahma Reddy Battula (xyao: rev 927577c87ca19e8b5b75722f78e2def6d9386576)
*
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java
Fix hdfs CLI usage message for namenode and zkfc
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-8551:
---
Attachment: HDFS-8551-003.patch
[~xyao] thanks for review..Attached the patch to fix the checkstyle comment
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551-003.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8551:
-
Component/s: documentation
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Components: documentation
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578364#comment-14578364
]
Xiaoyu Yao commented on HDFS-8551:
--
[~brahmareddy], Thanks for updating the patch. Looks like we still have one
more character to be moved to next line.
./hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java:268:
Line is longer than 80 characters (found 81).
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8337) httpfs doesn't work with creates from a jar with kerberos
[
https://issues.apache.org/jira/browse/HDFS-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578271#comment-14578271
]
Yongjun Zhang commented on HDFS-8337:
-
Hi [~tucu00],
Thanks again for your earlier help. Would you please help taking a look at rev
003? Thanks much!
httpfs doesn't work with creates from a jar with kerberos
-
Key: HDFS-8337
URL: https://issues.apache.org/jira/browse/HDFS-8337
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS, hdfs-client, security, webhdfs
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-8337.001.patch, HDFS-8337.002.patch,
HDFS-8337.003.patch
In a secure cluster, running a simple program:
{code}
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.security.*;
class Foo {
public static void main(String args[]) throws Exception {
FileSystem fs = FileSystem.get(new
java.net.URI(webhdfs://host:14000/), new Configuration());
System.out.println(fs.listStatus(new Path(/))[0]);
java.io.OutputStream os = fs.create(new Path(/tmp/foo));
os.write('a');
os.close();
}
}
{code}
Basically to access httpfs via webhdfs, the following exception is thrown:
{code}
[systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop
classpath):. Foo
15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded
Exception in thread main
org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException):
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
at Foo.main(Foo.java:7)
{code}
Thanks [~qwertymaniac] and [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8430) Erasure coding: update DFSClient.getFileChecksum() logic for stripe files
[
https://issues.apache.org/jira/browse/HDFS-8430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578279#comment-14578279
]
Tsz Wo Nicholas Sze commented on HDFS-8430:
---
Ideally, the checksums computed from a EC file and a replicated with the same
data should be the same.
Erasure coding: update DFSClient.getFileChecksum() logic for stripe files
-
Key: HDFS-8430
URL: https://issues.apache.org/jira/browse/HDFS-8430
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Walter Su
Assignee: Walter Su
HADOOP-3981 introduces a distributed file checksum algorithm. It's designed
for replicated block.
{{DFSClient.getFileChecksum()}} need some updates, so it can work for striped
block group.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8562) HDFS Performance is impacted by FileInputStream Finalizer
Yanping Wang created HDFS-8562:
--
Summary: HDFS Performance is impacted by FileInputStream Finalizer
Key: HDFS-8562
URL: https://issues.apache.org/jira/browse/HDFS-8562
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, HDFS, performance
Affects Versions: 2.5.0
Environment: Impact any application that uses HDFS
Reporter: Yanping Wang
While running HBase using HDFS as datanodes, we noticed excessive high GC pause
spikes. For example with jdk8 update 40 and G1 collector, we saw datanode GC
pauses spiked toward 160 milliseconds while they should be around 20
milliseconds.
We tracked down to GC logs and found those long GC pauses were devoted to
process high number of final references.
For example, this Young GC:
2715.501: [GC pause (G1 Evacuation Pause) (young) 0.1529017 secs]
2715.572: [SoftReference, 0 refs, 0.0001034 secs]
2715.572: [WeakReference, 0 refs, 0.123 secs]
2715.572: [FinalReference, 8292 refs, 0.0748194 secs]
2715.647: [PhantomReference, 0 refs, 160 refs, 0.0001333 secs]
2715.647: [JNI Weak Reference, 0.140 secs]
[Ref Proc: 122.3 ms]
[Eden: 910.0M(910.0M)-0.0B(911.0M) Survivors: 11.0M-10.0M Heap:
951.1M(1536.0M)-40.2M(1536.0M)]
[Times: user=0.47 sys=0.01, real=0.15 secs]
This young GC took 152.9 milliseconds STW pause, while spent 122.3 milliseconds
in Ref Proc, which processed 8292 FinalReference in 74.8 milliseconds plus some
overhead.
We used JFR and JMAP with Memory Analyzer to track down and found those
FinalReference were all from FileInputStream. We checked HDFS code and saw the
use of the FileInputStream in datanode:
https://apache.googlesource.com/hadoop-common/+/refs/heads/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/fsdataset/impl/MappableBlock.java
1. public static MappableBlock load(long length,
2. FileInputStream blockIn, FileInputStream metaIn,
3. String blockFileName) throws IOException {
4. MappableBlock mappableBlock = null;
5. MappedByteBuffer mmap = null;
6. FileChannel blockChannel = null;
7. try {
8. blockChannel = blockIn.getChannel();
9. if (blockChannel == null) {
10. throw new IOException(Block InputStream has no FileChannel.);
11. }
12. mmap = blockChannel.map(MapMode.READ_ONLY, 0, length);
13. NativeIO.POSIX.getCacheManipulator().mlock(blockFileName, mmap, length);
14. verifyChecksum(length, metaIn, blockChannel, blockFileName);
15. mappableBlock = new MappableBlock(mmap, length);
16. } finally {
17. IOUtils.closeQuietly(blockChannel);
18. if (mappableBlock == null) {
19. if (mmap != null) {
20. NativeIO.POSIX.munmap(mmap); // unmapping also unlocks
21. }
22. }
23. }
24. return mappableBlock;
25. }
We looked up
https://docs.oracle.com/javase/7/docs/api/java/io/FileInputStream.html and
http://hg.openjdk.java.net/jdk7/jdk7/jdk/file/23bdcede4e39/src/share/classes/java/io/FileInputStream.java
and noticed FileInputStream relies on the Finalizer to release its resource.
When a class that has a finalizer created, an entry for that class instance is
put on a queue in the JVM so the JVM knows it has a finalizer that needs to be
executed.
The current issue is: even with programmers do call close() after using
FileInputStream, its finalize() method will still be called. In other words,
still get the side effect of the FinalReference being registered at
FileInputStream allocation time, and also reference processing to reclaim the
FinalReference during GC (any GC solution has to deal with this).
We can imagine When running industry deployment HDFS, millions of files could
be opened and closed which resulted in a very large number of finalizers being
registered and subsequently being executed. That could cause very long GC
pause times.
We tried to use Files.newInputStream() to replace FileInputStream, but it was
clear we could not replace FileInputStream in
hdfs/server/datanode/fsdataset/impl/MappableBlock.java
We notified Oracle JVM team of this performance issue that impacting all Big
Data applications using HDFS. We recommended the proper fix in Java SE
FileInputStream. Because (1) it is really nothing wrong to use FileInputStream
in above datanode code, (2) as the object with a finalizer is registered with
finalizer list within the JVM at object allocation time, if someone makes an
explicit call to close or free the resources that are to be done in the
finalizer, then the finalizer should be pulled off the internal JVM’s finalizer
list. That will release the JVM from having to treat the object as special
because it has a finalizer, i.e. no need for GC to execute the finalizer as
part of Reference Processing.
As the java fix involves both JVM code and Java SE code, it might take time for
the full solution to
[jira] [Commented] (HDFS-8515) Abstract a DTP/2 HTTP/2 server
[
https://issues.apache.org/jira/browse/HDFS-8515?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578306#comment-14578306
]
Hadoop QA commented on HDFS-8515:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 6s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 36s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 53s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated
16 new checkstyle issues (total was 163, now 169). |
| {color:green}+1{color} | whitespace | 0m 1s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 34s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 20s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 17s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 161m 19s | Tests passed in hadoop-hdfs.
|
| | | 208m 19s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738478/HDFS-8515.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc2ed4a |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11283/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11283/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11283/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11283/console |
This message was automatically generated.
Abstract a DTP/2 HTTP/2 server
--
Key: HDFS-8515
URL: https://issues.apache.org/jira/browse/HDFS-8515
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Duo Zhang
Assignee: Duo Zhang
Attachments: HDFS-8515.patch
Discussed in HDFS-8471.
https://issues.apache.org/jira/browse/HDFS-8471?focusedCommentId=14568196page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14568196
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8552) Fix hdfs CLI usage message for namenode and zkfc
[
https://issues.apache.org/jira/browse/HDFS-8552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8552:
-
Resolution: Fixed
Fix Version/s: 2.8.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Thanks [~brahmareddy] for the contribution. I've committed the fix to trunk and
branch-2.
Fix hdfs CLI usage message for namenode and zkfc
Key: HDFS-8552
URL: https://issues.apache.org/jira/browse/HDFS-8552
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8552-002.patch, HDFS-8552.patch
The usage message from code does not match with the one documented.
1. java NameNode should be hdfs namenode
-2. Many of options are not documented.-
Usage message from code:
{code}
private static final String USAGE = Usage: java NameNode [
+ StartupOption.BACKUP.getName() + ] | \n\t[
+ StartupOption.CHECKPOINT.getName() + ] | \n\t[
+ StartupOption.FORMAT.getName() + [
+ StartupOption.CLUSTERID.getName() + cid ] [
+ StartupOption.FORCE.getName() + ] [
+ StartupOption.NONINTERACTIVE.getName() + ] ] | \n\t[
+ StartupOption.UPGRADE.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.UPGRADEONLY.getName() +
[ + StartupOption.CLUSTERID.getName() + cid] +
[ + StartupOption.RENAMERESERVED.getName() + k-v pairs] ] |
\n\t[
+ StartupOption.ROLLBACK.getName() + ] | \n\t[
+ StartupOption.ROLLINGUPGRADE.getName() +
+ RollingUpgradeStartupOption.getAllOptionString() + ] | \n\t[
+ StartupOption.IMPORT.getName() + ] | \n\t[
+ StartupOption.INITIALIZESHAREDEDITS.getName() + ] | \n\t[
+ StartupOption.BOOTSTRAPSTANDBY.getName() + ] | \n\t[
+ StartupOption.RECOVER.getName() + [
+ StartupOption.FORCE.getName() + ] ] | \n\t[
+ StartupOption.METADATAVERSION.getName() + ]
+ ];
{code}
Usage from document:
hdfs namenode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade |rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [-force] ] |
[-metadataVersion ]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8551) Fix hdfs datanode CLI usage message
[
https://issues.apache.org/jira/browse/HDFS-8551?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDFS-8551:
-
Component/s: (was: documentation)
Fix hdfs datanode CLI usage message
---
Key: HDFS-8551
URL: https://issues.apache.org/jira/browse/HDFS-8551
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.6.0
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Attachments: HDFS-8551-002.patch, HDFS-8551.patch
There are two issues with current hdfs datanode usage message below.
{code}
Usage: java DataNode [-regular | -rollback]
-regular : Normal DataNode startup (default).
-rollback: Rollback a standard or rolling upgrade.
Refer to HDFS documentation for the difference between standard
and rolling upgrades.
{code}
1. java DataNode should be hdfs datanode
2. rollingupgrace option is missing but it is document correctly in the
[link|http://hadoop.apache.org/docs/r2.7.0/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#datanode].
{code}
Usage: hdfs datanode [-regular | -rollback | -rollingupgrace rollback]
COMMAND_OPTIONDescription
-regular Normal datanode startup (default).
-rollback Rollback the datanode to the previous version. This should be
used after stopping the datanode and distributing the old hadoop version.
-rollingupgrade rollback Rollback a rolling upgrade operation.
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8553) Document hdfs class path options
[
https://issues.apache.org/jira/browse/HDFS-8553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578244#comment-14578244
]
Brahma Reddy Battula commented on HDFS-8553:
[~cnauroth] thanks for review and commit..
Document hdfs class path options
Key: HDFS-8553
URL: https://issues.apache.org/jira/browse/HDFS-8553
Project: Hadoop HDFS
Issue Type: Improvement
Components: documentation
Reporter: Xiaoyu Yao
Assignee: Brahma Reddy Battula
Fix For: 2.8.0
Attachments: HDFS-8553-002.patch, HDFS-8553.patch
--global, --jar options are not documented.
{code}
$ hdfs classpath --help
classpath [--glob|--jar path|-h|--help] :
Prints the classpath needed to get the Hadoop jar and the required
libraries.
Options:
--glob expand wildcards
--jar path write classpath as manifest in jar named path
-h, --help print help
{code}
current document:
{code}
User Commands
Commands useful for users of a hadoop cluster.
classpath
Usage: hdfs classpath
Prints the class path needed to get the Hadoop jar and the required libraries
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578291#comment-14578291
]
Kai Sasaki commented on HDFS-8494:
--
Thanks all for sharing inspiring ideas and thoughts! I think removing
hard-coded chunk-size requires more discussion and seems to have some
dependencies to solve.
So in this stage, we can assume chunk sizes used in EC cluster are the same to
{{HdfsConstants}} one.
{quote}
I think the basic idea is that we'd better decouple the logic of the blocks,
which belong to the storage level, from the files.
{quote}
I agree with [~jingzhao]. In order to remove hard-coded value, it might be
necessary to discuss more about the self-explained data structure in all cases.
Can we move this JIRA to under HDFS-8030 or HDFS-8031?
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8337) httpfs doesn't work with creates from a jar with kerberos
[
https://issues.apache.org/jira/browse/HDFS-8337?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578356#comment-14578356
]
Alejandro Abdelnur commented on HDFS-8337:
--
[~yzhangal], thanks for looking into this. After digging a bit in the code, I
think the intended way for this to work is that HTTPFS authentication filter
should do the following in its {{getConfiguration()}} method (KMS does exactly
this):
{code}
String authType = props.getProperty(AUTH_TYPE);
if (authType.equals(PseudoAuthenticationHandler.TYPE)) {
props.setProperty(AUTH_TYPE,
PseudoDelegationTokenAuthenticationHandler.class.getName());
} else if (authType.equals(KerberosAuthenticationHandler.TYPE)) {
props.setProperty(AUTH_TYPE,
KerberosDelegationTokenAuthenticationHandler.class.getName());
}
props.setProperty(DelegationTokenAuthenticationHandler.TOKEN_KIND,
WebHdfsConstants.WEBHDFS_TOKEN_KIND.toString());
{code}
httpfs doesn't work with creates from a jar with kerberos
-
Key: HDFS-8337
URL: https://issues.apache.org/jira/browse/HDFS-8337
Project: Hadoop HDFS
Issue Type: Bug
Components: HDFS, hdfs-client, security, webhdfs
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-8337.001.patch, HDFS-8337.002.patch,
HDFS-8337.003.patch
In a secure cluster, running a simple program:
{code}
import org.apache.hadoop.conf.*;
import org.apache.hadoop.fs.*;
import org.apache.hadoop.security.*;
class Foo {
public static void main(String args[]) throws Exception {
FileSystem fs = FileSystem.get(new
java.net.URI(webhdfs://host:14000/), new Configuration());
System.out.println(fs.listStatus(new Path(/))[0]);
java.io.OutputStream os = fs.create(new Path(/tmp/foo));
os.write('a');
os.close();
}
}
{code}
Basically to access httpfs via webhdfs, the following exception is thrown:
{code}
[systest@yj52s ~]$ /usr/java/jdk1.7.0_67-cloudera/bin/java -cp $(hadoop
classpath):. Foo
15/05/06 23:51:38 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded
Exception in thread main
org.apache.hadoop.ipc.RemoteException(com.sun.jersey.api.ParamException$QueryParamException):
java.lang.IllegalArgumentException: No enum constant
org.apache.hadoop.fs.http.client.HttpFSFileSystem.Operation.GETDELEGATIONTOKEN
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:163)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:354)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$200(WebHdfsFileSystem.java:91)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:608)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:1299)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getDelegationToken(WebHdfsFileSystem.java:237)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getAuthParameters(WebHdfsFileSystem.java:423)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toUrl(WebHdfsFileSystem.java:444)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractFsPathRunner.getUrl(WebHdfsFileSystem.java:691)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:603)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
at Foo.main(Foo.java:7)
{code}
Thanks [~qwertymaniac] and [~caseyjbrotherton] for reporting the issue.
--
This message was sent by Atlassian
[jira] [Commented] (HDFS-8499) Merge BlockInfoUnderConstruction into trunk
[
https://issues.apache.org/jira/browse/HDFS-8499?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578307#comment-14578307
]
Hadoop QA commented on HDFS-8499:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 18m 16s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 10 new or modified test files. |
| {color:green}+1{color} | javac | 7m 46s | There were no new javac warning
messages. |
| {color:red}-1{color} | javadoc | 9m 54s | The applied patch generated 3
additional warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 14s | The applied patch generated
30 new checkstyle issues (total was 1136, now 1143). |
| {color:green}+1{color} | whitespace | 0m 6s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 33s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 19s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:green}+1{color} | hdfs tests | 162m 3s | Tests passed in hadoop-hdfs.
|
| | | 209m 28s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738480/HDFS-8499.03.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / fc2ed4a |
| javadoc |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/diffJavadocWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf904.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11282/console |
This message was automatically generated.
Merge BlockInfoUnderConstruction into trunk
---
Key: HDFS-8499
URL: https://issues.apache.org/jira/browse/HDFS-8499
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.7.0
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Attachments: HDFS-8499.00.patch, HDFS-8499.01.patch,
HDFS-8499.02.patch, HDFS-8499.03.patch, HDFS-8499.UCFeature.patch
In HDFS-7285 branch, the {{BlockInfoUnderConstruction}} interface provides a
common abstraction for striped and contiguous UC blocks. This JIRA aims to
merge it to trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7576) TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes fails in Java 8 build
[
https://issues.apache.org/jira/browse/HDFS-7576?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14578329#comment-14578329
]
Xiaoyu Yao commented on HDFS-7576:
--
I saw the same test failed but with a different stack from this
https://builds.apache.org/job/PreCommit-HDFS-Build/11273/artifact/patchprocess/testrun_hadoop-hdfs.txt
today.
{code}
Tests run: 8, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 84.396 sec
FAILURE! - in org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
testFailoverRightBeforeCommitSynchronization(org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover)
Time elapsed: 2.029 sec ERROR!
java.io.IOException: Cannot obtain block length for
LocatedBlock{BP-316490051-67.195.81.148-1433795731197:blk_1073741825_1001;
getBlockSize()=2048; corrupt=false; offset=0;
locs=[DatanodeInfoWithStorage[127.0.0.1:59303,DS-1e89e854-057a-4e6a-8fb4-4f5ee9966e66,DISK],
DatanodeInfoWithStorage[127.0.0.1:58885,DS-a1d1439d-3e34-4ca1-b03e-e1aff5b873ea,DISK],
DatanodeInfoWithStorage[127.0.0.1:44884,DS-ec633801-467c-4f8e-b827-fcbd04dea5d7,DISK]]}
at
org.apache.hadoop.hdfs.DFSInputStream.readBlockLength(DFSInputStream.java:394)
at
org.apache.hadoop.hdfs.DFSInputStream.fetchLocatedBlocksAndGetLastBlockLength(DFSInputStream.java:336)
at
org.apache.hadoop.hdfs.DFSInputStream.openInfo(DFSInputStream.java:272)
at org.apache.hadoop.hdfs.DFSInputStream.init(DFSInputStream.java:263)
at org.apache.hadoop.hdfs.DFSClient.open(DFSClient.java:1184)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:308)
at
org.apache.hadoop.hdfs.DistributedFileSystem$3.doCall(DistributedFileSystem.java:304)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.open(DistributedFileSystem.java:304)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:768)
at
org.apache.hadoop.hdfs.DFSTestUtil.getFirstBlock(DFSTestUtil.java:747)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:353)
{code}
TestPipelinesFailover#testFailoverRightBeforeCommitSynchronization sometimes
fails in Java 8 build
--
Key: HDFS-7576
URL: https://issues.apache.org/jira/browse/HDFS-7576
Project: Hadoop HDFS
Issue Type: Test
Reporter: Ted Yu
Priority: Minor
From https://builds.apache.org/job/Hadoop-Hdfs-trunk-Java8/54/ :
{code}
REGRESSION:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization
Error Message:
test timed out after 3 milliseconds
Stack Trace:
java.lang.Exception: test timed out after 3 milliseconds
at sun.misc.Unsafe.park(Native Method)
at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)
at
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)
at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)
at
org.apache.hadoop.test.GenericTestUtils$DelayAnswer.waitForCall(GenericTestUtils.java:226)
at
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover.testFailoverRightBeforeCommitSynchronization(TestPipelinesFailover.java:386)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576713#comment-14576713
]
Kai Zheng commented on HDFS-8450:
-
The latest patch looks good to me. +1 pending on Jenkins build. Better if
[~vinayrpet] could also confirm.
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch,
HDFS-8450-HDFS-7285-10.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8450:
---
Attachment: (was: HDFS-8450-HDFS-7285-10.patch)
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
[ https://issues.apache.org/jira/browse/HDFS-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8556: Status: Patch Available (was: Open) Erasure Coding: Fix usage of 'createZone' - Key: HDFS-8556 URL: https://issues.apache.org/jira/browse/HDFS-8556 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Minor Attachments: HDFS-8556-HDFS-7285-01.patch Current usage for '-createZone' does not include 'cellSize' in the syntax But it shows in the detailed options. Add 'cellSize' also to one-line usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
[ https://issues.apache.org/jira/browse/HDFS-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8556: Attachment: (was: HDFS-8556-HDFS-7582-01.patch) Erasure Coding: Fix usage of 'createZone' - Key: HDFS-8556 URL: https://issues.apache.org/jira/browse/HDFS-8556 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Minor Attachments: HDFS-8556-HDFS-7285-01.patch Current usage for '-createZone' does not include 'cellSize' in the syntax But it shows in the detailed options. Add 'cellSize' also to one-line usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
[ https://issues.apache.org/jira/browse/HDFS-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8556: Attachment: HDFS-8556-HDFS-7285-01.patch Attached patch with correct name Erasure Coding: Fix usage of 'createZone' - Key: HDFS-8556 URL: https://issues.apache.org/jira/browse/HDFS-8556 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Minor Attachments: HDFS-8556-HDFS-7285-01.patch Current usage for '-createZone' does not include 'cellSize' in the syntax But it shows in the detailed options. Add 'cellSize' also to one-line usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8433) blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil
[ https://issues.apache.org/jira/browse/HDFS-8433?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Walter Su updated HDFS-8433: Attachment: HDFS-8433.01.patch Uploaded 01 patch. Delete retry logic of Inputstream. testRead() pass half part. Fully read case passes. Retry case fails. Please run the test you'll know. blockToken is not set in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil -- Key: HDFS-8433 URL: https://issues.apache.org/jira/browse/HDFS-8433 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo Nicholas Sze Assignee: Walter Su Attachments: HDFS-8433.00.patch, HDFS-8433.01.patch The blockToken provided in LocatedStripedBlock is not used to create LocatedBlock in constructInternalBlock and parseStripedBlockGroup in StripedBlockUtil. We should also add ec tests with security on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576683#comment-14576683
]
Kai Zheng commented on HDFS-8494:
-
Comments for the latest patch:
1. To avoid the overhead of {{getErasureCodingZone}} for each
{{StripedDataStreamer}} in the following change, please pass
{{ErasureCodingZone}} in the constructor of StripedDataStreamer, sharing the
same instance of ErasureCodingZone for all the needed streamers.
{code}
void putLoactedBlocks(LocatedBlock lb) throws IOException {
+ErasureCodingZone ecZone = dfsClient.getErasureCodingZone(src);
if (LOG.isDebugEnabled()) {
LOG.debug(Obtained block group + lb);
}
LocatedBlock[] blocks = StripedBlockUtil.parseStripedBlockGroup(
(LocatedStripedBlock)lb,
-BLOCK_STRIPED_CELL_SIZE, NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
+ecZone.getCellSize(), NUM_DATA_BLOCKS, NUM_PARITY_BLOCKS);
{code}
2. In balancer/Dispatcher, need to investigate how to remove the hard-coded
value. I would open separate JIRA for this as follow-on.
{code}
- HdfsConstants.BLOCK_STRIPED_CELL_SIZE, dataBlockNum, idxInGroup);
+ HdfsConstants.DEFAULT_BLOCK_STRIPED_CELL_SIZE, dataBlockNum,
idxInGroup);
{code}
3. Please avoid the 2nd call to {{getCellSize}}.
{code}
+ int cellSize = getCellSize(sblock);
+ if (cellSize 0) {
+dataBlockNum = (short) Math.min(dataBlockNum,
+(sblock.getNumBytes() - 1) / getCellSize(sblock) + 1);
+ }
{code}
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8494) Remove hard-coded chunk size in favor of ECZone
[
https://issues.apache.org/jira/browse/HDFS-8494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576708#comment-14576708
]
Walter Su commented on HDFS-8494:
-
bq. In balancer/Dispatcher, need to investigate how to remove the hard-coded
value.
It's easy to pass cellSize from BlockInfoStriped to Dispatcher. But the
question is BlockInfoStriped doesn't have it.
You have to get it by: BlockInfoStriped -- getBlockCollection -- getZone --
getCellSize. A bit complicated, isn't it?
I think BlockInfoStriped needs to keep cellSize.
Remove hard-coded chunk size in favor of ECZone
---
Key: HDFS-8494
URL: https://issues.apache.org/jira/browse/HDFS-8494
Project: Hadoop HDFS
Issue Type: Sub-task
Affects Versions: HDFS-7285
Reporter: Kai Sasaki
Assignee: Kai Sasaki
Fix For: HDFS-7285
Attachments: HDFS-8494-HDFS-7285-01.patch,
HDFS-8494-HDFS-7285-02.patch
It is necessary to remove hard-coded values inside NameNode configured in
{{HdfsConstants}}. In this JIRA, we can remove {{chunkSize}} gracefully in
favor of HDFS-8375.
Because {{cellSize}} is now originally stored only in {{ErasureCodingZone}},
{{BlockInfoStriped}} can receive {{cellSize}} in addition to {{ECSchema}}
when its initialization.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576660#comment-14576660
]
Vinayakumar B commented on HDFS-8450:
-
One more nit. Not exactly for this patch. But could go with this. In
ErasureCodingZoneManager
It would be better if all exceptions includes the {{src}} in messages.
Ex:
{code} throw new IOException(Attempt to create an erasure coding zone +
for a file.);{code}
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8450:
---
Attachment: HDFS-8450-HDFS-7285-10.patch
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch,
HDFS-8450-HDFS-7285-10.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576695#comment-14576695 ] Vincent.Wei commented on HDFS-7285: --- Hi all I am a new comer , I want to know if I can add the HDFS-7285-initial-PoC.patch on the hadoop v2.2.0 ? Thanks . Erasure Coding Support inside HDFS -- Key: HDFS-7285 URL: https://issues.apache.org/jira/browse/HDFS-7285 Project: Hadoop HDFS Issue Type: New Feature Reporter: Weihua Jiang Assignee: Zhe Zhang Attachments: ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch, HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf, fsimage-analysis-20150105.pdf Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data. Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back. We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7285) Erasure Coding Support inside HDFS
[ https://issues.apache.org/jira/browse/HDFS-7285?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576704#comment-14576704 ] Vinayakumar B commented on HDFS-7285: - Hi [~Vincent.Wei], thanks for looking here. bq.I am a new comer , I want to know if I can add the HDFS-7285-initial-PoC.patch on the hadoop v2.2.0 ? That patch was still in development phase at that time and it was based on trunk code. I think lot has changed from v2.2.0. Current progress is really good towards the completion of the Phase I of the EC feature. Hope feature will be available soon in trunk/branch-2. Erasure Coding Support inside HDFS -- Key: HDFS-7285 URL: https://issues.apache.org/jira/browse/HDFS-7285 Project: Hadoop HDFS Issue Type: New Feature Reporter: Weihua Jiang Assignee: Zhe Zhang Attachments: ECAnalyzer.py, ECParser.py, HDFS-7285-initial-PoC.patch, HDFSErasureCodingDesign-20141028.pdf, HDFSErasureCodingDesign-20141217.pdf, HDFSErasureCodingDesign-20150204.pdf, HDFSErasureCodingDesign-20150206.pdf, fsimage-analysis-20150105.pdf Erasure Coding (EC) can greatly reduce the storage overhead without sacrifice of data reliability, comparing to the existing HDFS 3-replica approach. For example, if we use a 10+4 Reed Solomon coding, we can allow loss of 4 blocks, with storage overhead only being 40%. This makes EC a quite attractive alternative for big data storage, particularly for cold data. Facebook had a related open source project called HDFS-RAID. It used to be one of the contribute packages in HDFS but had been removed since Hadoop 2.0 for maintain reason. The drawbacks are: 1) it is on top of HDFS and depends on MapReduce to do encoding and decoding tasks; 2) it can only be used for cold files that are intended not to be appended anymore; 3) the pure Java EC coding implementation is extremely slow in practical use. Due to these, it might not be a good idea to just bring HDFS-RAID back. We (Intel and Cloudera) are working on a design to build EC into HDFS that gets rid of any external dependencies, makes it self-contained and independently maintained. This design lays the EC feature on the storage type support and considers compatible with existing HDFS features like caching, snapshot, encryption, high availability and etc. This design will also support different EC coding schemes, implementations and policies for different deployment scenarios. By utilizing advanced libraries (e.g. Intel ISA-L library), an implementation can greatly improve the performance of EC encoding/decoding and makes the EC solution even more attractive. We will post the design document soon. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
[ https://issues.apache.org/jira/browse/HDFS-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B updated HDFS-8556: Attachment: HDFS-8556-HDFS-7582-01.patch Attached the patch for simple change Erasure Coding: Fix usage of 'createZone' - Key: HDFS-8556 URL: https://issues.apache.org/jira/browse/HDFS-8556 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Minor Attachments: HDFS-8556-HDFS-7582-01.patch Current usage for '-createZone' does not include 'cellSize' in the syntax But it shows in the detailed options. Add 'cellSize' also to one-line usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576733#comment-14576733
]
Rakesh R commented on HDFS-8450:
Thanks a lot [~drankye] and [~vinayrpet] for the reviews. Vinay, the latest
patch addresses your comments too. Please review when you get a chance!
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch,
HDFS-8450-HDFS-7285-10.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Rakesh R updated HDFS-8450:
---
Attachment: HDFS-8450-HDFS-7285-10.patch
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch,
HDFS-8450-HDFS-7285-10.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
Vinayakumar B created HDFS-8556: --- Summary: Erasure Coding: Fix usage of 'createZone' Key: HDFS-8556 URL: https://issues.apache.org/jira/browse/HDFS-8556 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Vinayakumar B Assignee: Vinayakumar B Priority: Minor Current usage for '-createZone' does not include 'cellSize' in the syntax But it shows in the detailed options. Add 'cellSize' also to one-line usage. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6232) OfflineEditsViewer throws a NPE on edits containing ACL modifications
[
https://issues.apache.org/jira/browse/HDFS-6232?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kanaka kumar avvaru updated HDFS-6232:
--
Assignee: Akira AJISAKA (was: kanaka kumar avvaru)
OfflineEditsViewer throws a NPE on edits containing ACL modifications
-
Key: HDFS-6232
URL: https://issues.apache.org/jira/browse/HDFS-6232
Project: Hadoop HDFS
Issue Type: Bug
Components: tools
Affects Versions: 3.0.0, 2.4.0
Reporter: Stephen Chu
Assignee: Akira AJISAKA
Fix For: 2.4.1
Attachments: HDFS-6232.2.patch, HDFS-6232.patch
The OfflineEditsViewer using the XML parser will through a NPE when using an
edit with a SET_ACL op.
{code}
[root@hdfs-nfs current]# hdfs oev -i
edits_001-007 -o fsedits.out
14/04/10 14:14:18 ERROR offlineEditsViewer.OfflineEditsBinaryLoader: Got
RuntimeException at position 505
Encountered exception. Exiting: null
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.util.XMLUtils.mangleXmlString(XMLUtils.java:122)
at org.apache.hadoop.hdfs.util.XMLUtils.addSaxString(XMLUtils.java:193)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.appendAclEntriesToXml(FSEditLogOp.java:4085)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.access$3300(FSEditLogOp.java:132)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp$SetAclOp.toXml(FSEditLogOp.java:3528)
at
org.apache.hadoop.hdfs.server.namenode.FSEditLogOp.outputToXml(FSEditLogOp.java:3928)
at
org.apache.hadoop.hdfs.tools.offlineEditsViewer.XmlEditsVisitor.visitOp(XmlEditsVisitor.java:116)
at
org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsBinaryLoader.loadEdits(OfflineEditsBinaryLoader.java:80)
at
org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.go(OfflineEditsViewer.java:142)
at
org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.run(OfflineEditsViewer.java:228)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.hdfs.tools.offlineEditsViewer.OfflineEditsViewer.main(OfflineEditsViewer.java:237)
[root@hdfs-nfs current]#
{code}
This is reproducible by setting an acl on a file and then running the OEV on
the editsinprogress file.
The stats and binary parsers run OK.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8556) Erasure Coding: Fix usage of 'createZone'
[
https://issues.apache.org/jira/browse/HDFS-8556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576794#comment-14576794
]
Hadoop QA commented on HDFS-8556:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:red}-1{color} | pre-patch | 15m 31s | Findbugs (version ) appears to
be broken on HDFS-7285. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 51s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:red}-1{color} | release audit | 0m 15s | The applied patch generated
1 release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 38s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 24s | The patch appears to introduce 4
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 21m 32s | Tests failed in hadoop-hdfs. |
| | | 64m 30s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestReplication |
| | hadoop.hdfs.TestLeaseRecovery |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotManager |
| | hadoop.hdfs.TestFileAppend4 |
| | hadoop.hdfs.TestRead |
| | hadoop.hdfs.server.namenode.TestFSPermissionChecker |
| | hadoop.hdfs.server.namenode.TestNameNodeRetryCacheMetrics |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshot |
| | hadoop.hdfs.server.namenode.TestSecondaryWebUi |
| | hadoop.hdfs.server.namenode.TestSaveNamespace |
| | hadoop.hdfs.qjournal.client.TestEpochsAreUnique |
| | hadoop.hdfs.server.namenode.TestNNStorageRetentionFunctional |
| | hadoop.hdfs.TestPread |
| | hadoop.hdfs.qjournal.server.TestJournal |
| | hadoop.hdfs.tools.TestGetGroups |
| | hadoop.hdfs.server.namenode.TestFavoredNodesEndToEnd |
| | hadoop.hdfs.server.namenode.ha.TestHAStateTransitions |
| | hadoop.hdfs.server.datanode.TestRefreshNamenodes |
| | hadoop.hdfs.TestHdfsAdmin |
| | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.datanode.TestDataNodeVolumeFailureReporting |
| | hadoop.hdfs.server.datanode.TestBlockHasMultipleReplicasOnSameDN |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotFileLength |
| | hadoop.hdfs.server.namenode.ha.TestLossyRetryInvocationHandler |
| | hadoop.hdfs.TestClientReportBadBlock |
| | hadoop.hdfs.TestSafeMode |
| | hadoop.hdfs.server.namenode.TestNamenodeCapacityReport |
| | hadoop.hdfs.server.namenode.TestNamenodeRetryCache |
| | hadoop.hdfs.TestParallelShortCircuitRead |
| | hadoop.hdfs.server.namenode.TestFSEditLogLoader |
| | hadoop.hdfs.server.namenode.TestAuditLogger |
| | hadoop.hdfs.TestFSInputChecker |
| | hadoop.hdfs.server.blockmanagement.TestUnderReplicatedBlocks |
| | hadoop.hdfs.TestDatanodeRegistration |
| | hadoop.hdfs.tools.TestDebugAdmin |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistPolicy |
| | hadoop.hdfs.server.blockmanagement.TestDatanodeManager |
| | hadoop.hdfs.qjournal.TestNNWithQJM |
| | hadoop.hdfs.TestSetrepIncreasing |
| | hadoop.hdfs.server.datanode.TestDataNodeMetrics |
| | hadoop.hdfs.TestDatanodeReport |
| | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.TestDFSShellGenericOptions |
| | hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistFiles |
| | hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup |
| |
hadoop.hdfs.server.datanode.fsdataset.impl.TestLazyPersistReplicaPlacement |
| | hadoop.hdfs.server.namenode.TestNameNodeRpcServer |
| | hadoop.hdfs.TestFileAppendRestart |
| | hadoop.hdfs.server.namenode.TestSecondaryNameNodeUpgrade |
| | hadoop.cli.TestErasureCodingCLI |
| | hadoop.hdfs.TestMultiThreadedHflush |
| | hadoop.hdfs.TestParallelRead |
| | hadoop.hdfs.TestWriteReadStripedFile |
| | hadoop.hdfs.server.namenode.snapshot.TestSetQuotaWithSnapshot |
| | hadoop.hdfs.server.namenode.ha.TestHAFsck |
| | hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots |
| | hadoop.hdfs.TestEncryptionZonesWithKMS |
| | hadoop.hdfs.tools.TestStoragePolicyCommands |
| | hadoop.hdfs.TestDFSRemove |
| |
[jira] [Commented] (HDFS-8450) Erasure Coding: Consolidate erasure coding zone related implementation into a single class
[
https://issues.apache.org/jira/browse/HDFS-8450?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576835#comment-14576835
]
Vinayakumar B commented on HDFS-8450:
-
{{fsd.getFSNamesystem().checkOperation(OperationCategory.READ);}}
This check is not mandatorily required in all places wherever
{{FSDirErasureCodingOp}} methods are called.
{{checkOperation}} is applicable only for the RPC calls. And if its already
done after acquiring the lock, then no need to do again.
All other cases which can happen only on ActiveNamenode after aquiring lock,
{{checkOperation}} call not required.
Following are the places where this check is not required in the patch.
1. {{FSDirStatAndListingOp.java}}
2. {{FSDirWriteFileOp.java}}
3. {{FSEditLogLoader}}
4. {{FSNamesystem#commitOrCompleteLastBlock(..)}} as this comes somewhere
inside in the call heirarchy, before reaching here check already done.
5. {{NamenodeFsck.java}} as in StandbyNameNode call will not reach this point,
if READ is disabled.
As though IMO, all these except #3, wont create any problem, but it would be
unnecessary. #3 might create problem in SNN while loading edits.
+1 once these are addressed.
Erasure Coding: Consolidate erasure coding zone related implementation into a
single class
--
Key: HDFS-8450
URL: https://issues.apache.org/jira/browse/HDFS-8450
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Rakesh R
Assignee: Rakesh R
Attachments: HDFS-8450-FYI.patch, HDFS-8450-HDFS-7285-00.patch,
HDFS-8450-HDFS-7285-01.patch, HDFS-8450-HDFS-7285-02.patch,
HDFS-8450-HDFS-7285-03.patch, HDFS-8450-HDFS-7285-04.patch,
HDFS-8450-HDFS-7285-05.patch, HDFS-8450-HDFS-7285-07.patch,
HDFS-8450-HDFS-7285-08.patch, HDFS-8450-HDFS-7285-09.patch,
HDFS-8450-HDFS-7285-10.patch
The idea is to follow the same pattern suggested by HDFS-7416. It is good to
consolidate all the erasure coding zone related implementations of
{{FSNamesystem}}. Here, proposing {{FSDirErasureCodingZoneOp}} class to have
functions to perform related erasure coding zone operations.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576781#comment-14576781
]
Hadoop QA commented on HDFS-8540:
-
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 49s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 1 new or modified test files. |
| {color:green}+1{color} | javac | 7m 31s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 39s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 22s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated 2
new checkstyle issues (total was 18, now 20). |
| {color:red}-1{color} | whitespace | 0m 0s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 36s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:red}-1{color} | findbugs | 3m 17s | The patch appears to introduce 2
new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | native | 3m 15s | Pre-build of native portion |
| {color:red}-1{color} | hdfs tests | 161m 44s | Tests failed in hadoop-hdfs. |
| | | 208m 6s | |
\\
\\
|| Reason || Tests ||
| FindBugs | module:hadoop-hdfs |
| Failed unit tests | hadoop.hdfs.TestAppendSnapshotTruncate |
| | hadoop.hdfs.server.mover.TestStorageMover |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12738157/HDFS-8540.patch |
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / a6cb489 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/diffcheckstylehadoop-hdfs.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/whitespace.txt
|
| Findbugs warnings |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
|
| hadoop-hdfs test log |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/artifact/patchprocess/testrun_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf909.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/11265/console |
This message was automatically generated.
Mover should exit with NO_MOVE_BLOCK if no block can be moved
-
Key: HDFS-8540
URL: https://issues.apache.org/jira/browse/HDFS-8540
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Reporter: Tsz Wo Nicholas Sze
Assignee: surendra singh lilhore
Attachments: HDFS-8540.patch
When there are files not satisfying their storage policy and no move is
possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
The bug seems in the following code. When StorageTypeDiff is not empty and
scheduleMoves4Block return false, it does not update hasRemaining. Also,
there is no indication of No block can be moved for the entire iteration.
{code}
//Mover.processFile(..)
if (!diff.removeOverlap(true)) {
if (scheduleMoves4Block(diff, lb, ecSchema)) {
hasRemaining |= (diff.existing.size() 1
diff.expected.size() 1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-8540) Mover should exit with NO_MOVE_BLOCK if no block can be moved
[
https://issues.apache.org/jira/browse/HDFS-8540?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14576979#comment-14576979
]
Vinayakumar B commented on HDFS-8540:
-
Thanks [~surendrasingh] for the patch.
Idea looks good.
Instead of creating separate instances of Result everytime while processing
recursively, why dont use Only one instance thoughout and update the same
result? Anyway you are updating the current result always with the returned
result.
Mover should exit with NO_MOVE_BLOCK if no block can be moved
-
Key: HDFS-8540
URL: https://issues.apache.org/jira/browse/HDFS-8540
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer mover
Reporter: Tsz Wo Nicholas Sze
Assignee: surendra singh lilhore
Attachments: HDFS-8540.patch
When there are files not satisfying their storage policy and no move is
possible, Mover exits with SUCCESS. It should exit with NO_MOVE_BLOCK.
The bug seems in the following code. When StorageTypeDiff is not empty and
scheduleMoves4Block return false, it does not update hasRemaining. Also,
there is no indication of No block can be moved for the entire iteration.
{code}
//Mover.processFile(..)
if (!diff.removeOverlap(true)) {
if (scheduleMoves4Block(diff, lb, ecSchema)) {
hasRemaining |= (diff.existing.size() 1
diff.expected.size() 1);
}
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
