[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[ https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037044#comment-14037044 ] Zesheng Wu commented on HDFS-6507: -- You mean that we should revert the change to ProxyAndInfo and remove addresses in logs messages also? > Improve DFSAdmin to support HA cluster better > - > > Key: HDFS-6507 > URL: https://issues.apache.org/jira/browse/HDFS-6507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch, > HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch, > HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch > > > Currently, the commands supported in DFSAdmin can be classified into three > categories according to the protocol used: > 1. ClientProtocol > Commands in this category generally implement by calling the corresponding > function of the DFSClient class, and will call the corresponding remote > implementation function at the NN side finally. At the NN side, all these > operations are classified into five categories: UNCHECKED, READ, WRITE, > CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only > allows UNCHECKED operations. In the current implementation of DFSClient, it > will connect one NN first, if the first NN is not Active and the operation is > not allowed, it will failover to the second NN. So here comes the problem, > some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage, > refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as > UNCHECKED operations, and when executing these commands in the DFSAdmin > command line, they will be sent to a definite NN, no matter it is Active or > Standby. This may result in two problems: > a. If the first tried NN is standby, and the operation takes effect only on > Standby NN, which is not the expected result. > b. If the operation needs to take effect on both NN, but it takes effect on > only one NN. In the future, when there is a NN failover, there may have > problems. > Here I propose the following improvements: > a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL > operations, we should classify it clearly. > b. If the command can not be classified as one of the above four operations, > or if the command needs to take effect on both NN, we should send the request > to both Active and Standby NNs. > 2. Refresh protocols: RefreshAuthorizationPolicyProtocol, > RefreshUserMappingsProtocol, RefreshUserMappingsProtocol, > RefreshCallQueueProtocol > Commands in this category, including refreshServiceAcl, > refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and > refreshCallQueue, are implemented by creating a corresponding RPC proxy and > sending the request to remote NN. In the current implementation, these > requests will be sent to a definite NN, no matter it is Active or Standby. > Here I propose that we sent these requests to both NNs. > 3. ClientDatanodeProtocol > Commands in this category are handled correctly, no need to improve. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[
https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037041#comment-14037041
]
Vinayakumar B commented on HDFS-6507:
-
Hi [~wuzesheng],
I just realized one more thing
In {{NameNodeProxies.ProxyAndInfo}} we need to set the address only while
creating the Non-HA proxies.
because In case of HA, it will set to ":8020" which is not correct
and its not used also.
This also can be mentioned somewhere in the comment.
So better not set the address for HA proxyinfo.
> Improve DFSAdmin to support HA cluster better
> -
>
> Key: HDFS-6507
> URL: https://issues.apache.org/jira/browse/HDFS-6507
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch,
> HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch,
> HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch
>
>
> Currently, the commands supported in DFSAdmin can be classified into three
> categories according to the protocol used:
> 1. ClientProtocol
> Commands in this category generally implement by calling the corresponding
> function of the DFSClient class, and will call the corresponding remote
> implementation function at the NN side finally. At the NN side, all these
> operations are classified into five categories: UNCHECKED, READ, WRITE,
> CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only
> allows UNCHECKED operations. In the current implementation of DFSClient, it
> will connect one NN first, if the first NN is not Active and the operation is
> not allowed, it will failover to the second NN. So here comes the problem,
> some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage,
> refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as
> UNCHECKED operations, and when executing these commands in the DFSAdmin
> command line, they will be sent to a definite NN, no matter it is Active or
> Standby. This may result in two problems:
> a. If the first tried NN is standby, and the operation takes effect only on
> Standby NN, which is not the expected result.
> b. If the operation needs to take effect on both NN, but it takes effect on
> only one NN. In the future, when there is a NN failover, there may have
> problems.
> Here I propose the following improvements:
> a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL
> operations, we should classify it clearly.
> b. If the command can not be classified as one of the above four operations,
> or if the command needs to take effect on both NN, we should send the request
> to both Active and Standby NNs.
> 2. Refresh protocols: RefreshAuthorizationPolicyProtocol,
> RefreshUserMappingsProtocol, RefreshUserMappingsProtocol,
> RefreshCallQueueProtocol
> Commands in this category, including refreshServiceAcl,
> refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and
> refreshCallQueue, are implemented by creating a corresponding RPC proxy and
> sending the request to remote NN. In the current implementation, these
> requests will be sent to a definite NN, no matter it is Active or Standby.
> Here I propose that we sent these requests to both NNs.
> 3. ClientDatanodeProtocol
> Commands in this category are handled correctly, no need to improve.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[ https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037035#comment-14037035 ] Vinayakumar B commented on HDFS-6507: - Yes [~jingzhao], I also agree that metaSave and saveNamespace commands are very heavy. But as these are only DFSAdmin commands, we assume admin needs to know when to execute these commands. > Improve DFSAdmin to support HA cluster better > - > > Key: HDFS-6507 > URL: https://issues.apache.org/jira/browse/HDFS-6507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch, > HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch, > HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch > > > Currently, the commands supported in DFSAdmin can be classified into three > categories according to the protocol used: > 1. ClientProtocol > Commands in this category generally implement by calling the corresponding > function of the DFSClient class, and will call the corresponding remote > implementation function at the NN side finally. At the NN side, all these > operations are classified into five categories: UNCHECKED, READ, WRITE, > CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only > allows UNCHECKED operations. In the current implementation of DFSClient, it > will connect one NN first, if the first NN is not Active and the operation is > not allowed, it will failover to the second NN. So here comes the problem, > some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage, > refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as > UNCHECKED operations, and when executing these commands in the DFSAdmin > command line, they will be sent to a definite NN, no matter it is Active or > Standby. This may result in two problems: > a. If the first tried NN is standby, and the operation takes effect only on > Standby NN, which is not the expected result. > b. If the operation needs to take effect on both NN, but it takes effect on > only one NN. In the future, when there is a NN failover, there may have > problems. > Here I propose the following improvements: > a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL > operations, we should classify it clearly. > b. If the command can not be classified as one of the above four operations, > or if the command needs to take effect on both NN, we should send the request > to both Active and Standby NNs. > 2. Refresh protocols: RefreshAuthorizationPolicyProtocol, > RefreshUserMappingsProtocol, RefreshUserMappingsProtocol, > RefreshCallQueueProtocol > Commands in this category, including refreshServiceAcl, > refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and > refreshCallQueue, are implemented by creating a corresponding RPC proxy and > sending the request to remote NN. In the current implementation, these > requests will be sent to a definite NN, no matter it is Active or Standby. > Here I propose that we sent these requests to both NNs. > 3. ClientDatanodeProtocol > Commands in this category are handled correctly, no need to improve. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6430) HTTPFS - Implement XAttr support
[ https://issues.apache.org/jira/browse/HDFS-6430?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036983#comment-14036983 ] Alejandro Abdelnur commented on HDFS-6430: -- Nice job Yi. A couple of minor NITs: * PBHelper.java, this seems like a false change * HttpFSUtils.java, new createURL(), we should URLEncoder.encode() the name of the multivalue parameter as well. Are testing the case of a FileSystem that does not support xAttrs? > HTTPFS - Implement XAttr support > > > Key: HDFS-6430 > URL: https://issues.apache.org/jira/browse/HDFS-6430 > Project: Hadoop HDFS > Issue Type: Task >Affects Versions: 3.0.0 >Reporter: Yi Liu >Assignee: Yi Liu > Fix For: 3.0.0 > > Attachments: HDFS-6430.1.patch, HDFS-6430.2.patch, HDFS-6430.3.patch, > HDFS-6430.patch > > > Add xattr support to HttpFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[ https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036982#comment-14036982 ] Hudson commented on HDFS-6563: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5733 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5733/]) HDFS-6563. NameNode cannot save fsimage in certain circumstances when snapshots are in use. Contributed by Aaron T. Myers. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603712) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImageFormatPBINode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/snapshot/FileWithSnapshotFeature.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestSnapshotBlocksMap.java > NameNode cannot save fsimage in certain circumstances when snapshots are in > use > --- > > Key: HDFS-6563 > URL: https://issues.apache.org/jira/browse/HDFS-6563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, snapshots >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Fix For: 2.5.0 > > Attachments: HDFS-6563.patch > > > Checkpoints will start to fail and the NameNode will not be able to manually > saveNamespace if the following set of steps occurs: > # A zero-length file appears in a snapshot > # That file is later lengthened to include at least one block > # That file is subsequently deleted from the present file system but remains > in the snapshot > More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[
https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036977#comment-14036977
]
Hadoop QA commented on HDFS-6549:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651329/HDFS-6549.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-nfs:
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7173//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7173//console
This message is automatically generated.
> Add support for accessing the NFS gateway from the AIX NFS client
> -
>
> Key: HDFS-6549
> URL: https://issues.apache.org/jira/browse/HDFS-6549
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
> Attachments: HDFS-6549.patch, HDFS-6549.patch
>
>
> We've identified two issues when trying to access the HDFS NFS Gateway from
> an AIX NFS client:
> # In the case of COMMITs, the AIX NFS client will always send 4096, or a
> multiple of the page size, for the offset to be committed, even if fewer
> bytes than this have ever, or will ever, be written to the file. This will
> cause a write to a file from the AIX NFS client to hang on close unless the
> size of that file is a multiple of 4096.
> # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the
> same cookie verifier for a given directory seemingly forever after that
> directory is first accessed over NFS, instead of getting a new cookie
> verifier for every set of incremental readdir calls. This means that if a
> directory's mtime ever changes, the FS must be unmounted/remounted before
> readdir calls on that dir from AIX will ever succeed again.
> From my interpretation of RFC-1813, the NFS Gateway is in fact doing the
> correct thing in both cases, but we can introduce simple changes on the NFS
> Gateway side to be able to optionally work around these incompatibilities.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[ https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6563: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks again for the review, Jing. I've just committed this to trunk and branch-2. > NameNode cannot save fsimage in certain circumstances when snapshots are in > use > --- > > Key: HDFS-6563 > URL: https://issues.apache.org/jira/browse/HDFS-6563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, snapshots >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Fix For: 2.5.0 > > Attachments: HDFS-6563.patch > > > Checkpoints will start to fail and the NameNode will not be able to manually > saveNamespace if the following set of steps occurs: > # A zero-length file appears in a snapshot > # That file is later lengthened to include at least one block > # That file is subsequently deleted from the present file system but remains > in the snapshot > More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036966#comment-14036966
]
Hudson commented on HDFS-6480:
--
FAILURE: Integrated in Hadoop-trunk-Commit #5732 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5732/])
HDFS-6480. Move waitForReady() from FSDirectory to FSNamesystem. Contributed by
Haohui Mai. (wheat9:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603705)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/Checkpointer.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestCommitBlockSynchronization.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFsLimits.java
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.5.0
>
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch,
> HDFS-6480.005.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[
https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036960#comment-14036960
]
Hadoop QA commented on HDFS-6563:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651320/HDFS-6563.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7172//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7172//console
This message is automatically generated.
> NameNode cannot save fsimage in certain circumstances when snapshots are in
> use
> ---
>
> Key: HDFS-6563
> URL: https://issues.apache.org/jira/browse/HDFS-6563
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, snapshots
>Affects Versions: 2.4.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
>Priority: Critical
> Attachments: HDFS-6563.patch
>
>
> Checkpoints will start to fail and the NameNode will not be able to manually
> saveNamespace if the following set of steps occurs:
> # A zero-length file appears in a snapshot
> # That file is later lengthened to include at least one block
> # That file is subsequently deleted from the present file system but remains
> in the snapshot
> More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036951#comment-14036951 ] Alejandro Abdelnur commented on HDFS-6312: -- planning to commit this tomorrow around noon PST. > WebHdfs HA failover is broken on secure clusters > > > Key: HDFS-6312 > URL: https://issues.apache.org/jira/browse/HDFS-6312 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-6312.attempted.patch > > > When webhdfs does a failover, it blanks out the delegation token. This will > cause subsequent operations against the other NN to acquire a new token. > Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6557:
-
Attachment: HDFS-6557.000.patch
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6557.000.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6557:
-
Status: Patch Available (was: Open)
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6557.000.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6480:
-
Resolution: Fixed
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to trunk and branch-2. Thanks [~jingzhao] for the
review.
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Fix For: 2.5.0
>
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch,
> HDFS-6480.005.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6519) Document oiv_legacy command
[
https://issues.apache.org/jira/browse/HDFS-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036909#comment-14036909
]
Hadoop QA commented on HDFS-6519:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651296/HDFS-6519.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7171//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7171//console
This message is automatically generated.
> Document oiv_legacy command
> ---
>
> Key: HDFS-6519
> URL: https://issues.apache.org/jira/browse/HDFS-6519
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: documentation
>Affects Versions: 2.5.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
> Attachments: HDFS-6519.patch
>
>
> HDFS-6293 introduced oiv_legacy command.
> The usage of the command should be included in OfflineImageViewer.apt.vm.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6566) Separate public and private keys in DFSConfigKey
[ https://issues.apache.org/jira/browse/HDFS-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036899#comment-14036899 ] Suresh Srinivas commented on HDFS-6566: --- It may be good to segregate client and server side configs. [~stev...@iseran.com] Has made a case for annotating all the configs as public. We should explore that. > Separate public and private keys in DFSConfigKey > > > Key: HDFS-6566 > URL: https://issues.apache.org/jira/browse/HDFS-6566 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai > > DFSConfigKey contains both public and private keys. There is no clear > clarification on whether a key should be public and stable. > This jira proposes to separate the public and stable keys in a separate class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036898#comment-14036898 ] Yongjun Zhang commented on HDFS-6403: - HI [~ajisakaa], thanks for the review, really appreciate it! > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036888#comment-14036888
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651333/HDFS-6200.007.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color:red}. The patch appears to cause the build to
fail.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7174//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch,
> HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036825#comment-14036825
]
Hadoop QA commented on HDFS-6492:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651295/HDFS-6492.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7170//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7170//console
This message is automatically generated.
> Support create-time xattrs and atomically setting multiple xattrs
> -
>
> Key: HDFS-6492
> URL: https://issues.apache.org/jira/browse/HDFS-6492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch,
> HDFS-6492.003.patch
>
>
> Ongoing work in HDFS-6134 requires being able to set system namespace
> extended attributes at create and mkdir time, as well as being able to
> atomically set multiple xattrs at once. There's currently no need to expose
> this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036816#comment-14036816 ] Hudson commented on HDFS-6553: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5731 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5731/]) HDFS-6553. Add missing DeprecationDeltas for NFS Kerberos configurations. Contributed by Stephen Chu (brandonli: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603677) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfiguration.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Fix For: 2.5.0 > > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036814#comment-14036814
]
Hadoop QA commented on HDFS-6562:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651291/HDFS-6562.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7169//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7169//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7169//console
This message is automatically generated.
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036806#comment-14036806
]
Hadoop QA commented on HDFS-6480:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651290/HDFS-6480.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7168//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7168//console
This message is automatically generated.
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch,
> HDFS-6480.005.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036804#comment-14036804
]
Hadoop QA commented on HDFS-2856:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651284/HDFS-2856.2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7167//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7167//console
This message is automatically generated.
> Fix block protocol so that Datanodes don't require root or jsvc
> ---
>
> Key: HDFS-2856
> URL: https://issues.apache.org/jira/browse/HDFS-2856
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Owen O'Malley
>Assignee: Chris Nauroth
> Attachments: Datanode-Security-Design.pdf,
> Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
> HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.prototype.patch
>
>
> Since we send the block tokens unencrypted to the datanode, we currently
> start the datanode as root using jsvc and get a secure (< 1024) port.
> If we have the datanode generate a nonce and send it on the connection and
> the sends an hmac of the nonce back instead of the block token it won't
> reveal any secrets. Thus, we wouldn't require a secure port and would not
> require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6567) Clean up HdfsFileStatus
[
https://issues.apache.org/jira/browse/HDFS-6567?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6567:
-
Description: As suggested in HDFS-6200, the order of "public final" in
{{HdfsFileStatus}} is reversed. This jira proposes to fix the order and to make
the code more consistent. (was: As suggested in HDFS-6200, the order of
"public final" in {{HdfsFileStatus}} is reserved. This jira proposes to fix the
order and to make the code more consistent.)
> Clean up HdfsFileStatus
> ---
>
> Key: HDFS-6567
> URL: https://issues.apache.org/jira/browse/HDFS-6567
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Haohui Mai
>
> As suggested in HDFS-6200, the order of "public final" in {{HdfsFileStatus}}
> is reversed. This jira proposes to fix the order and to make the code more
> consistent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6200) Create a separate jar for hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-6200: - Attachment: HDFS-6200.007.patch > Create a separate jar for hdfs-client > - > > Key: HDFS-6200 > URL: https://issues.apache.org/jira/browse/HDFS-6200 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Haohui Mai >Assignee: Haohui Mai > Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch, > HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch, > HDFS-6200.005.patch, HDFS-6200.006.patch, HDFS-6200.007.patch > > > Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs > client. As discussed in the hdfs-dev mailing list > (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser), > downstream projects are forced to bring in additional dependency in order to > access hdfs. The additional dependency sometimes can be difficult to manage > for projects like Apache Falcon and Apache Oozie. > This jira proposes to create a new project, hadoop-hdfs-cliient, which > contains the client side of the hdfs code. Downstream projects can use this > jar instead of the hadoop-hdfs to avoid unnecessary dependency. > Note that it does not break the compatibility of downstream projects. This is > because old downstream projects implicitly depend on hadoop-hdfs-client > through the hadoop-hdfs jar. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6567) Clean up HdfsFileStatus
Haohui Mai created HDFS-6567:
Summary: Clean up HdfsFileStatus
Key: HDFS-6567
URL: https://issues.apache.org/jira/browse/HDFS-6567
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
As suggested in HDFS-6200, the order of "public final" in {{HdfsFileStatus}} is
reserved. This jira proposes to fix the order and to make the code more
consistent.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036750#comment-14036750 ] Brandon Li commented on HDFS-6549: -- +1 to the latest patch. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036751#comment-14036751
]
Haohui Mai commented on HDFS-6200:
--
{quote}
DFSConfigKeys should be public and contain those keys clients are expected to
use
this is a good time to switch to SLF4J for the logging here, and drop
commons-logging
HdfsFileStatus gets its "public final" declarations in the wrong order ...
again, this is a good time to fix it.
JsonUtilClient uses org.mortbay.util.ajax.JSON to parse the json. Thus should
be replaced by Jackson, so we don't need the mortbay libs on the classpath?
WebHdfsFileSystem also uses mortbay JSON for parsing
{quote}
It might be better to restrict this patch to moving the files only. I plan to
address them in separate jiras. Filed HDFS-6564, HDFS-6565, HDFS-6566, and
HDFS-6567 to track them.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch,
> HDFS-6200.005.patch, HDFS-6200.006.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6566) Separate public and private keys in DFSConfigKey
Haohui Mai created HDFS-6566: Summary: Separate public and private keys in DFSConfigKey Key: HDFS-6566 URL: https://issues.apache.org/jira/browse/HDFS-6566 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai DFSConfigKey contains both public and private keys. There is no clear clarification on whether a key should be public and stable. This jira proposes to separate the public and stable keys in a separate class. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6549: - Attachment: HDFS-6549.patch Great, thanks. Here's an updated patch which adds an explanation about why AIX compat mode should not be enabled in general. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6565) Use jackson instead jetty json in hdfs-client
Haohui Mai created HDFS-6565: Summary: Use jackson instead jetty json in hdfs-client Key: HDFS-6565 URL: https://issues.apache.org/jira/browse/HDFS-6565 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai hdfs-client should use Jackson instead of jetty to parse JSON. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6564) Use slf4j instead of common-logging in hdfs-client
Haohui Mai created HDFS-6564: Summary: Use slf4j instead of common-logging in hdfs-client Key: HDFS-6564 URL: https://issues.apache.org/jira/browse/HDFS-6564 Project: Hadoop HDFS Issue Type: Bug Reporter: Haohui Mai hdfs-client should depends on slf4j instead of common-logging. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6387) HDFS CLI admin tool for creating & deleting an encryption zone
[ https://issues.apache.org/jira/browse/HDFS-6387?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6387: --- Attachment: HDFS-6387.003.patch .003 has all of the tests fixed to run correctly after HDFS-6386 (the backend of EZs) was checked in. > HDFS CLI admin tool for creating & deleting an encryption zone > -- > > Key: HDFS-6387 > URL: https://issues.apache.org/jira/browse/HDFS-6387 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Attachments: HDFS-6387.002.patch, HDFS-6387.003.patch, > HDFS-6387.1.patch > > > CLI admin tool to create/delete an encryption zone in HDFS. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[ https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036725#comment-14036725 ] Aaron T. Myers commented on HDFS-6563: -- Thanks a lot for the quick review, Jing. > NameNode cannot save fsimage in certain circumstances when snapshots are in > use > --- > > Key: HDFS-6563 > URL: https://issues.apache.org/jira/browse/HDFS-6563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, snapshots >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Attachments: HDFS-6563.patch > > > Checkpoints will start to fail and the NameNode will not be able to manually > saveNamespace if the following set of steps occurs: > # A zero-length file appears in a snapshot > # That file is later lengthened to include at least one block > # That file is subsequently deleted from the present file system but remains > in the snapshot > More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[
https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036720#comment-14036720
]
Zesheng Wu commented on HDFS-6507:
--
Thanks for feedback [~jingzhao]
I still think that we should send requests to all NNs. If people thinks some
commands are heavy(just as you mentioned), they can use the {{-fs}} option to
specify a NN to do his operation.
As the current default behavior of the commands(setSafeMode/metaSave, etc)
doesn't meet our expectation(you also think so), we should correct it. The way
you suggested is just a compromise, and it's easier to be friendly for people,
by not easier to other tools who use the DFSAdmin shell.
How do you think?
> Improve DFSAdmin to support HA cluster better
> -
>
> Key: HDFS-6507
> URL: https://issues.apache.org/jira/browse/HDFS-6507
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: tools
>Affects Versions: 2.4.0
>Reporter: Zesheng Wu
>Assignee: Zesheng Wu
> Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch,
> HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch,
> HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch
>
>
> Currently, the commands supported in DFSAdmin can be classified into three
> categories according to the protocol used:
> 1. ClientProtocol
> Commands in this category generally implement by calling the corresponding
> function of the DFSClient class, and will call the corresponding remote
> implementation function at the NN side finally. At the NN side, all these
> operations are classified into five categories: UNCHECKED, READ, WRITE,
> CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only
> allows UNCHECKED operations. In the current implementation of DFSClient, it
> will connect one NN first, if the first NN is not Active and the operation is
> not allowed, it will failover to the second NN. So here comes the problem,
> some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage,
> refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as
> UNCHECKED operations, and when executing these commands in the DFSAdmin
> command line, they will be sent to a definite NN, no matter it is Active or
> Standby. This may result in two problems:
> a. If the first tried NN is standby, and the operation takes effect only on
> Standby NN, which is not the expected result.
> b. If the operation needs to take effect on both NN, but it takes effect on
> only one NN. In the future, when there is a NN failover, there may have
> problems.
> Here I propose the following improvements:
> a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL
> operations, we should classify it clearly.
> b. If the command can not be classified as one of the above four operations,
> or if the command needs to take effect on both NN, we should send the request
> to both Active and Standby NNs.
> 2. Refresh protocols: RefreshAuthorizationPolicyProtocol,
> RefreshUserMappingsProtocol, RefreshUserMappingsProtocol,
> RefreshCallQueueProtocol
> Commands in this category, including refreshServiceAcl,
> refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and
> refreshCallQueue, are implemented by creating a corresponding RPC proxy and
> sending the request to remote NN. In the current implementation, these
> requests will be sent to a definite NN, no matter it is Active or Standby.
> Here I propose that we sent these requests to both NNs.
> 3. ClientDatanodeProtocol
> Commands in this category are handled correctly, no need to improve.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[ https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6563: - Status: Patch Available (was: Open) > NameNode cannot save fsimage in certain circumstances when snapshots are in > use > --- > > Key: HDFS-6563 > URL: https://issues.apache.org/jira/browse/HDFS-6563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, snapshots >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Attachments: HDFS-6563.patch > > > Checkpoints will start to fail and the NameNode will not be able to manually > saveNamespace if the following set of steps occurs: > # A zero-length file appears in a snapshot > # That file is later lengthened to include at least one block > # That file is subsequently deleted from the present file system but remains > in the snapshot > More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[ https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036713#comment-14036713 ] Jing Zhao commented on HDFS-6563: - Thanks for the fix, [~atm]! +1 pending Jenkins. > NameNode cannot save fsimage in certain circumstances when snapshots are in > use > --- > > Key: HDFS-6563 > URL: https://issues.apache.org/jira/browse/HDFS-6563 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode, snapshots >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers >Priority: Critical > Attachments: HDFS-6563.patch > > > Checkpoints will start to fail and the NameNode will not be able to manually > saveNamespace if the following set of steps occurs: > # A zero-length file appears in a snapshot > # That file is later lengthened to include at least one block > # That file is subsequently deleted from the present file system but remains > in the snapshot > More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[
https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Aaron T. Myers updated HDFS-6563:
-
Attachment: HDFS-6563.patch
Attached patch addresses the issue in two different ways:
# Don't set the blocks to {{null}} when cleaning up those blocks, but instead
set them to {{BlockInfo.EMPTY_ARRAY}} as is done elsewhere in the code.
# Make the FSImage saving code be tolerant of {{null}} blocks on INodeFiles.
Either one of these by itself would be sufficient to address this issue, but I
figure it's a good idea to do both so as to better future-proof this code
against possible accidental goofs.
Please review.
> NameNode cannot save fsimage in certain circumstances when snapshots are in
> use
> ---
>
> Key: HDFS-6563
> URL: https://issues.apache.org/jira/browse/HDFS-6563
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, snapshots
>Affects Versions: 2.4.0
>Reporter: Aaron T. Myers
>Assignee: Aaron T. Myers
>Priority: Critical
> Attachments: HDFS-6563.patch
>
>
> Checkpoints will start to fail and the NameNode will not be able to manually
> saveNamespace if the following set of steps occurs:
> # A zero-length file appears in a snapshot
> # That file is later lengthened to include at least one block
> # That file is subsequently deleted from the present file system but remains
> in the snapshot
> More details in the first comment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
[
https://issues.apache.org/jira/browse/HDFS-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036702#comment-14036702
]
Aaron T. Myers commented on HDFS-6563:
--
I've filed this as critical for now, but if folks think this should be a
blocker I'm fine raising the priority.
Though the issue is fairly critical, the bug is fairly straightforward. In
{{FSImageFormatPBINode#save(OutputStream, INodeFile)}} we have the following
code:
{code}
for (Block block : n.getBlocks()) {
b.addBlocks(PBHelper.convert(block));
}
{code}
Perhaps not obviously, this assumes that {{n.getBlocks()}} will never return
{{null}}. However, this is possible in the above-described scenario because of
this code in {{FileWithSnapshotFeature#collectBlocksBeyondMax}}:
{code}
final BlockInfo[] newBlocks;
if (n == 0) {
newBlocks = null;
} else {
newBlocks = new BlockInfo[n];
System.arraycopy(oldBlocks, 0, newBlocks, 0, n);
}
// set new blocks
file.setBlocks(newBlocks);
{code}
When attempting to save an fsimage after this code has been run, errors like
the following will appear in the logs:
{noformat}
2014-06-18 16:55:11,295 ERROR namenode.FSImage (FSImage.java:run(988)) - Unable
to save image for
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name1
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(FSImageFormatPBINode.java:537)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(FSImageFormatPBINode.java:518)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.serializeINodeSection(FSImageFormatPBINode.java:491)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveInodes(FSImageFormatProtobuf.java:412)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveInternal(FSImageFormatProtobuf.java:457)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.save(FSImageFormatProtobuf.java:393)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:931)
at
org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:982)
at java.lang.Thread.run(Thread.java:724)
2014-06-18 16:55:11,295 ERROR namenode.FSImage (FSImage.java:run(988)) - Unable
to save image for
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name2
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(FSImageFormatPBINode.java:537)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.save(FSImageFormatPBINode.java:518)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatPBINode$Saver.serializeINodeSection(FSImageFormatPBINode.java:491)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveInodes(FSImageFormatProtobuf.java:412)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.saveInternal(FSImageFormatProtobuf.java:457)
at
org.apache.hadoop.hdfs.server.namenode.FSImageFormatProtobuf$Saver.save(FSImageFormatProtobuf.java:393)
at
org.apache.hadoop.hdfs.server.namenode.FSImage.saveFSImage(FSImage.java:931)
at
org.apache.hadoop.hdfs.server.namenode.FSImage$FSImageSaver.run(FSImage.java:982)
at java.lang.Thread.run(Thread.java:724)
2014-06-18 16:55:11,297 ERROR common.Storage
(NNStorage.java:reportErrorsOnDirectory(808)) - Error reported on storage
directory Storage Directory
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name1
2014-06-18 16:55:11,297 WARN common.Storage
(NNStorage.java:reportErrorsOnDirectory(813)) - About to remove corresponding
storage:
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name1
2014-06-18 16:55:11,297 ERROR common.Storage
(NNStorage.java:reportErrorsOnDirectory(808)) - Error reported on storage
directory Storage Directory
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name2
2014-06-18 16:55:11,297 WARN common.Storage
(NNStorage.java:reportErrorsOnDirectory(813)) - About to remove corresponding
storage:
/home/atm/src/apache/hadoop.git/src/hadoop-hdfs-project/hadoop-hdfs/build/test/data/dfs/name2
{noformat}
> NameNode cannot save fsimage in certain circumstances when snapshots are in
> use
> ---
>
> Key: HDFS-6563
> URL: https://issues.apache.org/jira/browse/HDFS-6563
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode, snapshots
>Affect
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036700#comment-14036700
]
Steve Loughran commented on HDFS-6200:
--
# DFSConfigKeys should be public and contain those keys clients are expected to
use
# this is a good time to switch to SLF4J for the logging here, and drop
commons-logging
# HdfsFileStatus gets its "public final" declarations in the wrong order ...
again, this is a good time to fix it.
# JsonUtilClient uses {{org.mortbay.util.ajax.JSON}} to parse the json. Thus
should be replaced by Jackson, so we don't need the mortbay libs on the
classpath?
# {{WebHdfsFileSystem}} also uses mortbay JSON for parsing
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch,
> HDFS-6200.005.patch, HDFS-6200.006.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6244) Make Trash Interval configurable for each of the namespaces
[ https://issues.apache.org/jira/browse/HDFS-6244?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036698#comment-14036698 ] Siqi Li commented on HDFS-6244: --- can anyone give me some review for this > Make Trash Interval configurable for each of the namespaces > --- > > Key: HDFS-6244 > URL: https://issues.apache.org/jira/browse/HDFS-6244 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.0.5-alpha >Reporter: Siqi Li >Assignee: Siqi Li > Attachments: HDFS-6244.v1.patch, HDFS-6244.v2.patch, > HDFS-6244.v3.patch > > > Somehow we need to avoid the cluster filling up. > One solution is to have a different trash policy per namespace. However, if > we can simply make the property configurable per namespace, then the same > config can be rolled everywhere and we'd be done. This seems simple enough. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6563) NameNode cannot save fsimage in certain circumstances when snapshots are in use
Aaron T. Myers created HDFS-6563: Summary: NameNode cannot save fsimage in certain circumstances when snapshots are in use Key: HDFS-6563 URL: https://issues.apache.org/jira/browse/HDFS-6563 Project: Hadoop HDFS Issue Type: Bug Components: namenode, snapshots Affects Versions: 2.4.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Priority: Critical Checkpoints will start to fail and the NameNode will not be able to manually saveNamespace if the following set of steps occurs: # A zero-length file appears in a snapshot # That file is later lengthened to include at least one block # That file is subsequently deleted from the present file system but remains in the snapshot More details in the first comment. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036683#comment-14036683 ] Stephen Chu commented on HDFS-6553: --- Thanks, Brandon :) > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Fix For: 2.5.0 > > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036681#comment-14036681 ] Brandon Li commented on HDFS-6553: -- I've committed the patch. Thanks! > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Fix For: 2.5.0 > > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6553: - Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Fix For: 2.5.0 > > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6553: - Fix Version/s: 2.5.0 > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Fix For: 2.5.0 > > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036670#comment-14036670 ] Brandon Li commented on HDFS-6553: -- Sure. > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036665#comment-14036665 ] Brandon Li commented on HDFS-6549: -- Yes. +1 once the doc is updated. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade
[
https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036627#comment-14036627
]
Akira AJISAKA commented on HDFS-6558:
-
The patch is just to fix the formatting, so new tests are not needed.
> Missing '\n' in the description of dfsadmin -rollingUpgrade
> ---
>
> Key: HDFS-6558
> URL: https://issues.apache.org/jira/browse/HDFS-6558
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Chen He
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-6558.patch
>
>
> In DFSAdmin.java, '\n' should be added at the end of the line
> {code}
> + " prepare: prepare a new rolling upgrade."
> {code}
> to clean up the following help message.
> {code}
> $ hdfs dfsadmin -help rollingUpgrade
> -rollingUpgrade []:
> query: query the current rolling upgrade status.
>prepare: prepare a new rolling upgrade. finalize: finalize the current
> rolling upgrade.
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade
[
https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036624#comment-14036624
]
Hadoop QA commented on HDFS-6558:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651231/HDFS-6558.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7165//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7165//console
This message is automatically generated.
> Missing '\n' in the description of dfsadmin -rollingUpgrade
> ---
>
> Key: HDFS-6558
> URL: https://issues.apache.org/jira/browse/HDFS-6558
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Chen He
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-6558.patch
>
>
> In DFSAdmin.java, '\n' should be added at the end of the line
> {code}
> + " prepare: prepare a new rolling upgrade."
> {code}
> to clean up the following help message.
> {code}
> $ hdfs dfsadmin -help rollingUpgrade
> -rollingUpgrade []:
> query: query the current rolling upgrade status.
>prepare: prepare a new rolling upgrade. finalize: finalize the current
> rolling upgrade.
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reassigned HDFS-6391: - Assignee: Andrew Wang (was: Charles Lamb) > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036617#comment-14036617
]
Alejandro Abdelnur commented on HDFS-6134:
--
[~lmccay], thanks for following up.
The proposed approach would be applicable using different cipher modes than CTR
(ie CBC and XTR, even GCM if we handle the offset correction changes because of
the GCM tag). In all cases, it would enable keeping the EZ keyVersion materials
unknown to HDFS and clients and exposing the DEK for the files being accessed
to the client accessing the file only.
In the case of CTR, the proposed approach also helps avoiding the IV reuse
scenario.
Trying to answer your questions:
bq. how does the keyprovider know what EZ key to use - is it the key that is
referenced by the keyVersionName?
{code}
// creates EDEK using specified version name
KeyVersion generateEncryptedKey(String versionName, byte[] iv)
// receives EDEK returns DEK using specified version name
KeyVersion decryptEncryptedKey(String versionName, byte[] iv, KeyVersion
encryptedKey)
{code}
The callers of both methods have the versionName at hand
bq. how do we key HDFS clients from asking for the EZ key - if it is stored by
the passed in keyVersionName?
The file iNode will store (EZ-keyVersionName, IV, EDEK), that info is passed to
the client on create()/open(). Using that info, the client can go to the
KeyProvider to get the DEK for the file.
bq. will this require special access control protection for EZ keys?
KeyProviders could implement special control. For example KMS allows, via ACLs,
to get KeyVersion without key material. This will effectively prevent EZ keys
from leaving the KMS.
bq. would the unique DEK be stored in the provider as well or only in the
extended attributes of the file? if stored in the provider what is the
keyVersionName for it?
the unique EDEKs (encrypted with the EZ keyversion) are not stored by the
KeyProvider but in the xAttr of the file.
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthÂcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036619#comment-14036619
]
Akira AJISAKA commented on HDFS-6559:
-
Now Jenkins says +1, there's no need to revert. Thanks [~arpitagarwal] for the
review!
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6200) Create a separate jar for hdfs-client
[
https://issues.apache.org/jira/browse/HDFS-6200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036611#comment-14036611
]
Hadoop QA commented on HDFS-6200:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651236/HDFS-6200.006.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs hadoop-hdfs-project/hadoop-hdfs-client
hadoop-hdfs-project/hadoop-hdfs/src/contrib/bkjournal:
org.apache.hadoop.hdfs.web.TestWebHDFSAcl
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7164//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7164//console
This message is automatically generated.
> Create a separate jar for hdfs-client
> -
>
> Key: HDFS-6200
> URL: https://issues.apache.org/jira/browse/HDFS-6200
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6200.000.patch, HDFS-6200.001.patch,
> HDFS-6200.002.patch, HDFS-6200.003.patch, HDFS-6200.004.patch,
> HDFS-6200.005.patch, HDFS-6200.006.patch
>
>
> Currently the hadoop-hdfs jar contain both the hdfs server and the hdfs
> client. As discussed in the hdfs-dev mailing list
> (http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-dev/201404.mbox/browser),
> downstream projects are forced to bring in additional dependency in order to
> access hdfs. The additional dependency sometimes can be difficult to manage
> for projects like Apache Falcon and Apache Oozie.
> This jira proposes to create a new project, hadoop-hdfs-cliient, which
> contains the client side of the hdfs code. Downstream projects can use this
> jar instead of the hadoop-hdfs to avoid unnecessary dependency.
> Note that it does not break the compatibility of downstream projects. This is
> because old downstream projects implicitly depend on hadoop-hdfs-client
> through the hadoop-hdfs jar.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6519) Document oiv_legacy command
[ https://issues.apache.org/jira/browse/HDFS-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-6519: Attachment: HDFS-6519.patch * Add a document of oiv_legacy in HdfsImageViewer.apt.vm. * Add a new property "dfs.namenode.legacy-oiv-image.dir" in hdfs-default.xml > Document oiv_legacy command > --- > > Key: HDFS-6519 > URL: https://issues.apache.org/jira/browse/HDFS-6519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.5.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HDFS-6519.patch > > > HDFS-6293 introduced oiv_legacy command. > The usage of the command should be included in OfflineImageViewer.apt.vm. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6519) Document oiv_legacy command
[ https://issues.apache.org/jira/browse/HDFS-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA updated HDFS-6519: Target Version/s: 2.5.0 Status: Patch Available (was: Open) > Document oiv_legacy command > --- > > Key: HDFS-6519 > URL: https://issues.apache.org/jira/browse/HDFS-6519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.5.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > Attachments: HDFS-6519.patch > > > HDFS-6293 introduced oiv_legacy command. > The usage of the command should be included in OfflineImageViewer.apt.vm. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6492: -- Attachment: HDFS-6492.003.patch Thanks for the reviews Charles and Colin, this patch addresses your review comments. > Support create-time xattrs and atomically setting multiple xattrs > - > > Key: HDFS-6492 > URL: https://issues.apache.org/jira/browse/HDFS-6492 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, > HDFS-6492.003.patch > > > Ongoing work in HDFS-6134 requires being able to set system namespace > extended attributes at create and mkdir time, as well as being able to > atomically set multiple xattrs at once. There's currently no need to expose > this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6562:
-
Attachment: HDFS-6562.000.patch
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6562:
-
Status: Patch Available (was: Open)
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6562) Refactor rename() in FSDirectory
Haohui Mai created HDFS-6562:
Summary: Refactor rename() in FSDirectory
Key: HDFS-6562
URL: https://issues.apache.org/jira/browse/HDFS-6562
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-6562.000.patch
Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
Both implementation shares quite a bit of common code.
This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6480:
-
Attachment: HDFS-6480.005.patch
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch,
> HDFS-6480.005.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036593#comment-14036593 ] Aaron T. Myers commented on HDFS-6549: -- All makes sense. So are you good with this patch if I update the NFS user doc to explain why AIX compatibility mode is not recommended in general? > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036589#comment-14036589 ] Brandon Li commented on HDFS-6549: -- Guess AIX NFS server doesn't put mtime in the verifier and uses it in a different way. :-) The patch should be able to fix the AIX NFS client problems. The only thing is that, the aix.compatibility.mode should not be recommended for non AIX client for the following reasons: 1. NFS client considers it's safe to clear the buffer cache as long as the data is committed. In aix.compatibility.mode, NFS could returns commit success even when the data is not received. 2. Many NFS clients restart directory listing once it encounters verifier mismatch error. In aix.compatibility.mode, client may not get the most recent directory content. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036583#comment-14036583
]
Hadoop QA commented on HDFS-6480:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651283/HDFS-6480.004.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7166//console
This message is automatically generated.
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-2856:
Attachment: HDFS-2856.2.patch
I'm attaching v2 of the patch. This has the following changes since last time:
# Added new tests for the balancer with SASL on DataTransferProtocol. This
helped me find a bug with the balancer passing around the incorrect datanode ID
(the source instead of the destination), so I fixed that.
# Removed TODO for inclusion of block pool ID in the SASL handshake. I already
include the token identifier, which contains the block pool ID as a component,
so it's not necessary to add block pool ID again.
# Removed the client-generated timestamp from the SASL handshake. The original
intention of the timestamp was to make it harder for a man in the middle to
replay the message. The server side would have checked elapsed time since the
timestamp and rejected the request if it was beyond a threshold. However, the
SASL DIGEST-MD5 handshake already protects against this, because the server
initiates a random challenge at the start of any new connection. It's highly
likely that the challenge will be unique across different connection attempts,
and thus a replayed message is highly likely to be rejected. The timestamp
wouldn't provide any additional benefit.
# Removed datanode ID from the SASL handshake. This had been intended to
protect against a man in the middle rerouting a message to a different
datanode. As described above, SASL DIGEST-MD5 already protects against this,
because the server issues a different challenge on each connection attempt.
The datanode ID wouldn't provide any additional benefit.
# Added code in {{DataNode#checkSecureConfig}} to check that when SASL is used
on DataTransferProtocol, SSL must also be used on HTTP. Plain HTTP wouldn't be
safe, because the client could write a delegation token query parameter onto
the socket without any authentication of the server. By requiring SSL, we
enforce that the server is authenticated before sending the delegation token.
> Fix block protocol so that Datanodes don't require root or jsvc
> ---
>
> Key: HDFS-2856
> URL: https://issues.apache.org/jira/browse/HDFS-2856
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, security
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Owen O'Malley
>Assignee: Chris Nauroth
> Attachments: Datanode-Security-Design.pdf,
> Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
> HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.prototype.patch
>
>
> Since we send the block tokens unencrypted to the datanode, we currently
> start the datanode as root using jsvc and get a secure (< 1024) port.
> If we have the datanode generate a nonce and send it on the connection and
> the sends an hmac of the nonce back instead of the block token it won't
> reveal any secrets. Thus, we wouldn't require a secure port and would not
> require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6386) HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HDFS-6386. Resolution: Fixed Target Version/s: fs-encryption (HADOOP-10150 and HDFS-6134) Committed to fs-encryption. Thanks Andrew. > HDFS Encryption Zones > - > > Key: HDFS-6386 > URL: https://issues.apache.org/jira/browse/HDFS-6386 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HDFS-6386.012.patch, HDFS-6386.013.patch, > HDFS-6386.014.patch, HDFS-6386.4.patch, HDFS-6386.5.patch, HDFS-6386.6.patch, > HDFS-6386.8.patch > > > Define the required security xAttributes for directories and files within an > encryption zone and how they propagate to children. Implement the logic to > create/delete encryption zones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6480:
-
Attachment: HDFS-6480.004.patch
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch, HDFS-6480.004.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036560#comment-14036560 ] Larry McCay commented on HDFS-6134: --- [~tucu00] - I realize that it is the first usecase - that doesn't make it the only one that we have in mind or in the works. The fact that you have run into an issue with the EZ key granularity while using the CTR mode is a problem with the usecase design not necessarily with the abstraction of key providers. The question is whether wrapped keys will be required by other usecases where either the key usage pattern or the encryption modes in use may not require them. Currently, the KeyProvider API doesn't do any encryption itself - I just want to make sure that adding the additional complexity and responsibility to this interface is really necessary. Additional questions: * how does the keyprovider know what EZ key to use - is it the key that is referenced by the keyVersionName? * how do we key HDFS clients from asking for the EZ key - if it is stored by the passed in keyVersionName? ** will this require special access control protection for EZ keys? * would the unique DEK be stored in the provider as well or only in the extended attributes of the file? ** if stored in the provider what is the keyVersionName for it? > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5711) Removing memory limitation of the Namenode by persisting Block - Block location mappings to disk.
[ https://issues.apache.org/jira/browse/HDFS-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5711: - Assignee: (was: Haohui Mai) > Removing memory limitation of the Namenode by persisting Block - Block > location mappings to disk. > - > > Key: HDFS-5711 > URL: https://issues.apache.org/jira/browse/HDFS-5711 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rohan Pasalkar >Priority: Minor > > This jira is to track changes to be made to remove HDFS name-node memory > limitation to hold block - block location mappings. > It is a known fact that the single Name-node architecture of HDFS has > scalability limits. The HDFS federation project alleviates this problem by > using horizontal scaling. This helps increase the throughput of metadata > operation and also the amount of data that can be stored in a Hadoop cluster. > The Name-node stores all the filesystem metadata in memory (even in the > federated architecture), the > Name-node design can be enhanced by persisting part of the metadata onto > secondary storage and retaining > the popular or recently accessed metadata information in main memory. This > design can benefit a HDFS deployment > which doesn't use federation but needs to store a large number of files or > large number of blocks. Lin Xiao from Hortonworks attempted a similar > project [1] in the Summer of 2013. They used LevelDB to persist the Namespace > information (i.e file and directory inode information). > A patch with this change is yet to be submitted to code base. We also intend > to use LevelDB to persist metadata, and plan to > provide a complete solution, by not just persisting the Namespace > information but also the Blocks Map onto secondary storage. > We did implement the basic prototype which stores the block-block location > mapping metadata to the persistent key-value store i.e. levelDB. Prototype > also maintains the in-memory cache of the recently used block-block location > mappings metadata. > References: > [1] Lin Xiao, Hortonworks, Removing Name-nodeâs memory limitation, HDFS-5389, > http://www.slideshare.net/ydn/hadoop-meetup-hug-august-2013-removing-the-namenodes-memory-limitation. > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6386) HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6386: --- Attachment: HDFS-6386.014.patch Thanks for the review Andrew. The .014 patch removes those args. > HDFS Encryption Zones > - > > Key: HDFS-6386 > URL: https://issues.apache.org/jira/browse/HDFS-6386 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HDFS-6386.012.patch, HDFS-6386.013.patch, > HDFS-6386.014.patch, HDFS-6386.4.patch, HDFS-6386.5.patch, HDFS-6386.6.patch, > HDFS-6386.8.patch > > > Define the required security xAttributes for directories and files within an > encryption zone and how they propagate to children. Implement the logic to > create/delete encryption zones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036547#comment-14036547 ] Stephen Chu commented on HDFS-6553: --- Thanks, [~brandonli]! Would you have time to commit? (not a committer) > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-5389) A Namenode that keeps only a part of the namespace in memory
[ https://issues.apache.org/jira/browse/HDFS-5389?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5389: Assignee: Haohui Mai > A Namenode that keeps only a part of the namespace in memory > > > Key: HDFS-5389 > URL: https://issues.apache.org/jira/browse/HDFS-5389 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Affects Versions: 0.23.1 >Reporter: Lin Xiao >Assignee: Haohui Mai >Priority: Minor > > *Background:* > Currently, the NN Keeps all its namespace in memory. This has had the benefit > that the NN code is very simple and, more importantly, helps the NN scale to > over 4.5K machines with 60K to 100K concurrently tasks. HDFS namespace can > be scaled currently using more Ram on the NN and/or using Federation which > scales both namespace and performance. The current federation implementation > does not allow renames across volumes without data copying but there are > proposals to remove that limitation. > *Motivation:* > Hadoop lets customers store huge amounts of data at very economical prices > and hence allows customers to store their data for several years. While most > customers perform analytics on recent data (last hour, day, week, months, > quarter, year), the ability to have five year old data online for analytics > is very attractive for many businesses. Although one can use larger RAM in a > NN and/or use Federation, it not really necessary to store the entire > namespace in memory since only the recent data is typically heavily accessed. > *Proposed Solution:* > Store a portion of the NN's namespace in memory- the "working set" of the > applications that are currently operating. LSM data structures are quite > appropriate for maintaining the full namespace in memory. One choice is > Google's LevelDB open-source implementation. > *Benefits:* > * Store larger namespaces without resorting to Federated namespace volumes. > * Complementary to NN Federated namespace volumes, indeed will allow a > single NN to easily store multiple larger volumes. > * Faster cold startup - the NN does not have read its full namespace before > responding to clients. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-5711) Removing memory limitation of the Namenode by persisting Block - Block location mappings to disk.
[ https://issues.apache.org/jira/browse/HDFS-5711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5711: Assignee: Haohui Mai > Removing memory limitation of the Namenode by persisting Block - Block > location mappings to disk. > - > > Key: HDFS-5711 > URL: https://issues.apache.org/jira/browse/HDFS-5711 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode >Reporter: Rohan Pasalkar >Assignee: Haohui Mai >Priority: Minor > > This jira is to track changes to be made to remove HDFS name-node memory > limitation to hold block - block location mappings. > It is a known fact that the single Name-node architecture of HDFS has > scalability limits. The HDFS federation project alleviates this problem by > using horizontal scaling. This helps increase the throughput of metadata > operation and also the amount of data that can be stored in a Hadoop cluster. > The Name-node stores all the filesystem metadata in memory (even in the > federated architecture), the > Name-node design can be enhanced by persisting part of the metadata onto > secondary storage and retaining > the popular or recently accessed metadata information in main memory. This > design can benefit a HDFS deployment > which doesn't use federation but needs to store a large number of files or > large number of blocks. Lin Xiao from Hortonworks attempted a similar > project [1] in the Summer of 2013. They used LevelDB to persist the Namespace > information (i.e file and directory inode information). > A patch with this change is yet to be submitted to code base. We also intend > to use LevelDB to persist metadata, and plan to > provide a complete solution, by not just persisting the Namespace > information but also the Blocks Map onto secondary storage. > We did implement the basic prototype which stores the block-block location > mapping metadata to the persistent key-value store i.e. levelDB. Prototype > also maintains the in-memory cache of the recently used block-block location > mappings metadata. > References: > [1] Lin Xiao, Hortonworks, Removing Name-nodeâs memory limitation, HDFS-5389, > http://www.slideshare.net/ydn/hadoop-meetup-hug-august-2013-removing-the-namenodes-memory-limitation. > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6519) Document oiv_legacy command
[ https://issues.apache.org/jira/browse/HDFS-6519?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HDFS-6519: --- Assignee: Akira AJISAKA > Document oiv_legacy command > --- > > Key: HDFS-6519 > URL: https://issues.apache.org/jira/browse/HDFS-6519 > Project: Hadoop HDFS > Issue Type: Improvement > Components: documentation >Affects Versions: 2.5.0 >Reporter: Akira AJISAKA >Assignee: Akira AJISAKA > > HDFS-6293 introduced oiv_legacy command. > The usage of the command should be included in OfflineImageViewer.apt.vm. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6386) HDFS Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6386?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036510#comment-14036510 ] Andrew Wang commented on HDFS-6386: --- Hi Charles, * logRetryCache is still being passed unnecessarily to FSDir methods +1 pending that though, thanks > HDFS Encryption Zones > - > > Key: HDFS-6386 > URL: https://issues.apache.org/jira/browse/HDFS-6386 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Charles Lamb > Fix For: fs-encryption (HADOOP-10150 and HDFS-6134) > > Attachments: HDFS-6386.012.patch, HDFS-6386.013.patch, > HDFS-6386.4.patch, HDFS-6386.5.patch, HDFS-6386.6.patch, HDFS-6386.8.patch > > > Define the required security xAttributes for directories and files within an > encryption zone and how they propagate to children. Implement the logic to > create/delete encryption zones. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[
https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036504#comment-14036504
]
Colin Patrick McCabe commented on HDFS-6492:
{code}
import java.util.*;
{code}
Shouldn't use wildcard includes (turn off this setting in your editor)
{code}
@VisibleForTesting
List filterINodeXAttrs(List existingXAttrs,
List filterXAttrs, List filteredXAttrs) {
{code}
{{filterXAttrs}} sounds confusingly similar to {{filteredXAttrs}}. Perhaps
rename these to {{toFilter}} and {{removed}}?
{{FSNamesystem#removeXAttr}}: we don't log the name of the xattr that was
removed? Seems like we should do this. Maybe in a separate JIRA, though.
+1 once these are addressed
> Support create-time xattrs and atomically setting multiple xattrs
> -
>
> Key: HDFS-6492
> URL: https://issues.apache.org/jira/browse/HDFS-6492
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 2.4.0
>Reporter: Andrew Wang
>Assignee: Andrew Wang
> Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch
>
>
> Ongoing work in HDFS-6134 requires being able to set system namespace
> extended attributes at create and mkdir time, as well as being able to
> atomically set multiple xattrs at once. There's currently no need to expose
> this functionality in the client API, so let's not unless we have to.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036474#comment-14036474
]
Hadoop QA commented on HDFS-6559:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651221/HDFS-6559.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+0 tests included{color}. The patch appears to be a
documentation patch that doesn't require tests.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7162//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7162//console
This message is automatically generated.
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6482) Use block ID-based block layout on datanodes
[
https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036465#comment-14036465
]
Hadoop QA commented on HDFS-6482:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651220/HDFS-6482.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 15 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDatanodeLayoutUpgrade
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7163//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7163//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7163//console
This message is automatically generated.
> Use block ID-based block layout on datanodes
>
>
> Key: HDFS-6482
> URL: https://issues.apache.org/jira/browse/HDFS-6482
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 2.5.0
>Reporter: James Thomas
>Assignee: James Thomas
> Attachments: HDFS-6482.1.patch, HDFS-6482.2.patch, HDFS-6482.3.patch,
> HDFS-6482.4.patch, HDFS-6482.5.patch, HDFS-6482.patch
>
>
> Right now blocks are placed into directories that are split into many
> subdirectories when capacity is reached. Instead we can use a block's ID to
> determine the path it should go in. This eliminates the need for the LDir
> data structure that facilitates the splitting of directories when they reach
> capacity as well as fields in ReplicaInfo that keep track of a replica's
> location.
> An extension of the work in HDFS-3290.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6418) Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
[
https://issues.apache.org/jira/browse/HDFS-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Steve Loughran updated HDFS-6418:
-
Assignee: Tsz Wo Nicholas Sze
> Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
> ---
>
> Key: HDFS-6418
> URL: https://issues.apache.org/jira/browse/HDFS-6418
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Steve Loughran
>Assignee: Tsz Wo Nicholas Sze
>
> Code i have that compiles against HADOOP 2.4 doesn't build against trunk as
> someone took away {{DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY}} -apparently in
> HDFS-6181.
> I know the name was obsolete, but anyone who has compiled code using that
> reference -rather than cutting and pasting in the string- is going to find
> their code doesn't work.
> More subtly: that will lead to a link exception trying to run that code on a
> 2.5+ cluster.
> This is a regression: the old names need to go back in, even if they refer to
> the new names and are marked as deprecated
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036446#comment-14036446 ] Aaron T. Myers commented on HDFS-6549: -- bq. Large directory may need multiple READDIR/READDIRPLUS requests. Many NFS client implementations send 0 as the verifier for the first READDIR/READDIRPLUS RPC request. In the AIX case, after it grabs a verifier from server it will always use the same verifier even in the initial READDIR/READIRPLUS request when it accesses the same directory again. Is my understanding correct? Correct, this is exactly what I've observed. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036443#comment-14036443 ] Brandon Li commented on HDFS-6549: -- This sometimes happens with some traditional NFS server/client implementations: when the NFS client does READDIR/READDIRPLUS with a huge directory which keeps changing, the READDIR/READDIRPLUS can never finish because the verifier changes and NFS client has to restart the listing request before it finishes. But based on what you described, the AIX client is a different case. Large directory may need multiple READDIR/READDIRPLUS requests. Many NFS client implementations send 0 as the verifier for the first READDIR/READDIRPLUS RPC request. In the AIX case, after it grabs a verifier from server it will always use the same verifier even in the initial READDIR/READIRPLUS request when it accesses the same directory again. Is my understanding correct? > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036437#comment-14036437
]
Hudson commented on HDFS-6559:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5729 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5729/])
HDFS-6559. Fix wrong option "dfsadmin -rollingUpgrade start" in the document.
(Contributed by Akira Ajisaka) (arp:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603638)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/xdoc/HdfsRollingUpgrade.xml
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036434#comment-14036434
]
Arpit Agarwal commented on HDFS-6559:
-
My bad, I missed waiting for the Jenkins +1.
Since this is a trivial documentation-only patch I won't revert it for now. In
the unlikely case there is a problem I'll back it out.
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6559:
Resolution: Fixed
Fix Version/s: 2.5.0
3.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I committed this to trunk and branch-2.
Thanks for the contribution [~ajisakaa].
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6559) Fix wrong option "dfsadmin -rollingUpgrade start" in the document
[
https://issues.apache.org/jira/browse/HDFS-6559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036419#comment-14036419
]
Arpit Agarwal commented on HDFS-6559:
-
+1
I will commit it shortly.
> Fix wrong option "dfsadmin -rollingUpgrade start" in the document
> -
>
> Key: HDFS-6559
> URL: https://issues.apache.org/jira/browse/HDFS-6559
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Akira AJISAKA
>Priority: Minor
> Labels: newbie
> Attachments: HDFS-6559.patch
>
>
> In HdfsRollingUpgrade.xml,
> {code}
> hdfs dfsadmin -rollingUpgrade
> {code}
> should be
> {code}
> hdfs dfsadmin -rollingUpgrade
>
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036417#comment-14036417
]
Jing Zhao commented on HDFS-6480:
-
The 003 patch looks pretty good to me. Two minors:
# Currently in FSNamesystem#renameTo, we call waitForLoadingFSImage right
before we call FSDirectory#renameTo. In that way, we may invoke
FSDirectory.resolvePath before we can guarantee that the fsdir has been fully
loaded.
# We need to update the following comment in several places:
{code}
- // check if we are ready to initialize replication queues
+ // check if we are imageLoaded to initialize replication queues
{code}
+1 after addressing the comments.
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6561) Byte array native checksumming on client side
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas reassigned HDFS-6561: -- Assignee: James Thomas > Byte array native checksumming on client side > - > > Key: HDFS-6561 > URL: https://issues.apache.org/jira/browse/HDFS-6561 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, performance >Reporter: James Thomas >Assignee: James Thomas > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6560) Byte array native checksumming on DN side
[ https://issues.apache.org/jira/browse/HDFS-6560?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Thomas reassigned HDFS-6560: -- Assignee: James Thomas > Byte array native checksumming on DN side > - > > Key: HDFS-6560 > URL: https://issues.apache.org/jira/browse/HDFS-6560 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, performance >Reporter: James Thomas >Assignee: James Thomas > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6560) Byte array native checksumming on DN side
James Thomas created HDFS-6560: -- Summary: Byte array native checksumming on DN side Key: HDFS-6560 URL: https://issues.apache.org/jira/browse/HDFS-6560 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Thomas -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6561) Byte array native checksumming on client side
James Thomas created HDFS-6561: -- Summary: Byte array native checksumming on client side Key: HDFS-6561 URL: https://issues.apache.org/jira/browse/HDFS-6561 Project: Hadoop HDFS Issue Type: Sub-task Reporter: James Thomas -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036370#comment-14036370
]
Hudson commented on HDFS-6439:
--
FAILURE: Integrated in Hadoop-trunk-Commit #5728 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5728/])
HDFS-6439. NFS should not reject NFS requests to the NULL procedure whether
port monitoring is enabled or not. Contributed by Brandon Li (brandonli:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603622)
*
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/nfs/nfs3/Nfs3Interface.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/main/java/org/apache/hadoop/oncrpc/RpcProgram.java
*
/hadoop/common/trunk/hadoop-common-project/hadoop-nfs/src/test/java/org/apache/hadoop/oncrpc/TestFrameDecoder.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfiguration.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/mount/RpcProgramMountd.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/Nfs3.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/TestReaddir.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestReaddir.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Fix For: 2.5.0
>
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6480) Move waitForReady() from FSDirectory to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036363#comment-14036363
]
Hadoop QA commented on HDFS-6480:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651198/HDFS-6480.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7161//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7161//console
This message is automatically generated.
> Move waitForReady() from FSDirectory to FSNamesystem
>
>
> Key: HDFS-6480
> URL: https://issues.apache.org/jira/browse/HDFS-6480
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6480.000.patch, HDFS-6480.001.patch,
> HDFS-6480.002.patch, HDFS-6480.003.patch
>
>
> Currently FSDirectory implements a barrier in {{waitForReady()}} /
> {{setReady()}} so that it only serve requests once the FSImage is fully
> loaded.
> As a part of the effort to evolve {{FSDirectory}} to a class which focuses on
> implementing the data structure of the namespace, this jira proposes to move
> the barrier one level higher to {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6439:
-
Affects Version/s: (was: 2.4.0)
2.5.0
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Fix For: 2.5.0
>
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6439:
-
Fix Version/s: 2.5.0
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Fix For: 2.5.0
>
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036356#comment-14036356
]
Brandon Li commented on HDFS-6439:
--
I've committed the patch. Thanks again, Aaron.
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Fix For: 2.5.0
>
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Li updated HDFS-6439:
-
Resolution: Fixed
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.5.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Fix For: 2.5.0
>
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036344#comment-14036344 ] Alejandro Abdelnur commented on HDFS-6134: -- [~lmccay], it is great that you agree that the proposed changes makes things more secure. Answering your bullets: #1. this is the first use case in Hadoop #2. no, the client does not have acces to the EZ keyVersion #3. the KeyProvider own the decryption of the DEK ,the client does not have the EZ key What is being proposed is the equivalent of KMIP's key-wrapping functionality. What makes the system more secure is the fact that the EZ keyVersion is never exposed to HDFS, nor HDFS clients. And the DEK for the file is never exposed to HDFS NN. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthÂcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6553) Add missing DeprecationDeltas for NFS Kerberos configurations
[ https://issues.apache.org/jira/browse/HDFS-6553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036332#comment-14036332 ] Brandon Li commented on HDFS-6553: -- +1 > Add missing DeprecationDeltas for NFS Kerberos configurations > - > > Key: HDFS-6553 > URL: https://issues.apache.org/jira/browse/HDFS-6553 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 3.0.0, 2.5.0 >Reporter: Stephen Chu >Assignee: Stephen Chu > Attachments: HDFS-6553.patch.001, HDFS-6553.patch.002 > > > HDFS-6056 (Clean up NFS config settings) improved NFS configuration naming > and added DeprecationDeltas to ensure that the old NFS configuration > properties could still be used. > It's currently missing DeprecationDeltas for _dfs.nfs.keytab.file_ and > _dfs.nfs.kerberos.principal_. > This patch adds those deltas so older configs for secure NFS can still work. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036310#comment-14036310
]
Aaron T. Myers commented on HDFS-6439:
--
Agreed. +1
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6439) NFS should not reject NFS requests to the NULL procedure whether port monitoring is enabled or not
[
https://issues.apache.org/jira/browse/HDFS-6439?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14036303#comment-14036303
]
Brandon Li commented on HDFS-6439:
--
The test failures are not introduced by this patch.
> NFS should not reject NFS requests to the NULL procedure whether port
> monitoring is enabled or not
> --
>
> Key: HDFS-6439
> URL: https://issues.apache.org/jira/browse/HDFS-6439
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: nfs
>Affects Versions: 2.4.0
>Reporter: Brandon Li
>Assignee: Aaron T. Myers
> Attachments: HDFS-6439.003.patch, HDFS-6439.004.patch,
> HDFS-6439.005.patch, HDFS-6439.patch, HDFS-6439.patch,
> linux-nfs-disallow-request-from-nonsecure-port.pcapng,
> mount-nfs-requests.pcapng
>
>
> As discussed in HDFS-6406, this JIRA is to track the follow update:
> 1. Port monitoring is the feature name with traditional NFS server and we may
> want to make the config property (along with related variable
> allowInsecurePorts) something as dfs.nfs.port.monitoring.
> 2 . According to RFC2623 (http://www.rfc-editor.org/rfc/rfc2623.txt):
> {quote}Whether port monitoring is enabled or not, NFS servers SHOULD NOT
> reject NFS requests to the NULL procedure (procedure number 0). See
> subsection 2.3.1, "NULL procedure" for a complete explanation. {quote}
> I do notice that NFS clients (most time) send mount NULL and nfs NULL from
> non-privileged port. If we deny NULL call in mountd or nfs server, the client
> can't mount the export even as user root.
> 3. it would be nice to have the user guide updated for the port monitoring
> feature.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6558) Missing '\n' in the description of dfsadmin -rollingUpgrade
[
https://issues.apache.org/jira/browse/HDFS-6558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Akira AJISAKA updated HDFS-6558:
Assignee: Chen He
Target Version/s: 2.5.0
Status: Patch Available (was: Open)
> Missing '\n' in the description of dfsadmin -rollingUpgrade
> ---
>
> Key: HDFS-6558
> URL: https://issues.apache.org/jira/browse/HDFS-6558
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 2.4.0
>Reporter: Akira AJISAKA
>Assignee: Chen He
>Priority: Trivial
> Labels: newbie
> Attachments: HDFS-6558.patch
>
>
> In DFSAdmin.java, '\n' should be added at the end of the line
> {code}
> + " prepare: prepare a new rolling upgrade."
> {code}
> to clean up the following help message.
> {code}
> $ hdfs dfsadmin -help rollingUpgrade
> -rollingUpgrade []:
> query: query the current rolling upgrade status.
>prepare: prepare a new rolling upgrade. finalize: finalize the current
> rolling upgrade.
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
