[jira] [Updated] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-4667: Attachment: HDFS-4667.004.patch Thanks for the review, Nicholas! Update the patch to address your comments. Also added another two unit tests. I will create another jira to move and refactor SnapshotDiffInfo. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, > HDFS-4667.v1.patch, getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038457#comment-14038457
]
Hadoop QA commented on HDFS-6562:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651588/HDFS-6562.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7186//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7186//console
This message is automatically generated.
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[ https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038448#comment-14038448 ] Vinayakumar B commented on HDFS-6507: - Hi all, If no other comments from anyone before tomorrow, I will go ahead and commit this. Thanks. > Improve DFSAdmin to support HA cluster better > - > > Key: HDFS-6507 > URL: https://issues.apache.org/jira/browse/HDFS-6507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch, > HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch, > HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch > > > Currently, the commands supported in DFSAdmin can be classified into three > categories according to the protocol used: > 1. ClientProtocol > Commands in this category generally implement by calling the corresponding > function of the DFSClient class, and will call the corresponding remote > implementation function at the NN side finally. At the NN side, all these > operations are classified into five categories: UNCHECKED, READ, WRITE, > CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only > allows UNCHECKED operations. In the current implementation of DFSClient, it > will connect one NN first, if the first NN is not Active and the operation is > not allowed, it will failover to the second NN. So here comes the problem, > some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage, > refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as > UNCHECKED operations, and when executing these commands in the DFSAdmin > command line, they will be sent to a definite NN, no matter it is Active or > Standby. This may result in two problems: > a. If the first tried NN is standby, and the operation takes effect only on > Standby NN, which is not the expected result. > b. If the operation needs to take effect on both NN, but it takes effect on > only one NN. In the future, when there is a NN failover, there may have > problems. > Here I propose the following improvements: > a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL > operations, we should classify it clearly. > b. If the command can not be classified as one of the above four operations, > or if the command needs to take effect on both NN, we should send the request > to both Active and Standby NNs. > 2. Refresh protocols: RefreshAuthorizationPolicyProtocol, > RefreshUserMappingsProtocol, RefreshUserMappingsProtocol, > RefreshCallQueueProtocol > Commands in this category, including refreshServiceAcl, > refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and > refreshCallQueue, are implemented by creating a corresponding RPC proxy and > sending the request to remote NN. In the current implementation, these > requests will be sent to a definite NN, no matter it is Active or Standby. > Here I propose that we sent these requests to both NNs. > 3. ClientDatanodeProtocol > Commands in this category are handled correctly, no need to improve. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Issue Comment Deleted] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-6134:
--
Comment: was deleted
(was: {quote}
Vanilla distcp will just work with transparent encryption. Data will be
decrypted on read and encrypted on write, assuming both source and target are
in encrypted zones.
The proposal on changing distcp is to enable a second use used case, copy data
from one cluster to another without having to decrypt/encrypt the data while
doing the copy. This is useful when doing copies for disaster recovery, hdfs
admins could do the copy without having to have access to the encryption keys.
{quote}
Agree.)
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: ConceptualDesignProposal_2014-06-19.pdf,
> HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[
https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038443#comment-14038443
]
Hadoop QA commented on HDFS-6572:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651578/HDFS-6572.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestCrcCorruption
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7185//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7185//console
This message is automatically generated.
> Add an option to the NameNode that prints the software and on-disk image
> versions
> -
>
> Key: HDFS-6572
> URL: https://issues.apache.org/jira/browse/HDFS-6572
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Reporter: Charles Lamb
>Assignee: Charles Lamb
>Priority: Minor
> Attachments: HDFS-6572.001.patch
>
>
> The HDFS namenode should have a startup option that prints the metadata
> versions of both the software and the on-disk version. This will be useful
> for debugging certain situations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-6510) WebHdfs clients clears the delegation token on retry (for HA), thus failing retry requests
[ https://issues.apache.org/jira/browse/HDFS-6510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang resolved HDFS-6510. - Resolution: Duplicate > WebHdfs clients clears the delegation token on retry (for HA), thus failing > retry requests > -- > > Key: HDFS-6510 > URL: https://issues.apache.org/jira/browse/HDFS-6510 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > > In WebHdfs clients connected to a HA HDFS service, when a failure (that is > inferred as a failover) happens and retry is attempted, the delegation token > previously initialized is cleared. For token based auth, this causes all the > subsequent retries to fail due to auth errors. > See "delegationToken = null" in method WebHdfsFileSystem.resetStateToFailOver. > This issue would not only show up on failover, but happens more commonly when > the first NN specified in the config is not reachable. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038438#comment-14038438 ] Hudson commented on HDFS-6403: -- FAILURE: Integrated in Hadoop-trunk-Commit #5741 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5741/]) HDFS-6403. Add metrics for log warnings reported by JVM pauses. Contributed by Yongjun Zhang. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604074) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Metrics.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/source/TestJvmMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.5.0 > > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038437#comment-14038437 ] Alejandro Abdelnur commented on HDFS-6134: -- [~hitliuyi], on the why we need the IV in the proposed new methods. We need an IV because EDEKs are transient from the KeyProvider perspective. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-19.pdf, > HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038422#comment-14038422
]
Yi Liu commented on HDFS-6134:
--
{quote}
Vanilla distcp will just work with transparent encryption. Data will be
decrypted on read and encrypted on write, assuming both source and target are
in encrypted zones.
The proposal on changing distcp is to enable a second use used case, copy data
from one cluster to another without having to decrypt/encrypt the data while
doing the copy. This is useful when doing copies for disaster recovery, hdfs
admins could do the copy without having to have access to the encryption keys.
{quote}
Agree.
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: ConceptualDesignProposal_2014-06-19.pdf,
> HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038415#comment-14038415
]
Yi Liu commented on HDFS-6134:
--
*1.*
{quote}
• 6. Because of #5, a possible attack would be to scan #1 files for IVs where
the 8 higher bytes match. Then, fast-forward them to a common counter point
(assuming files are long enough), then you’ll have more than one cypher-text
using the same encryption key and the same IV. The chances of this are 1/2^64,
but in cryptographic terms this is considered a high chance.
{quote}
CTR attack is not by finding two cipher-text using same encryption key and same
IV, it’s by ability of constructing file and make it encrypted by same Data key
and IV. The principle is as following: (Suppose we have two plain texts: P1 and
P2; Cipher texts: C1 and C2)
{code}
P1 XOR F(Key, IV) = C1
P2 XOR F(Key, IV) = C2
{code}
Absolutely C1 and C2 are known. To guess P2, if we just know IV (we can’t know
key, it’s secret), but don’t know P1, then we can’t get P2. But if we can
constructed P1 (then we know it), and make it encrypted by same Key and IV,
then we can easily know P2 through:
{code}
P2 = C2 XOR F(Key, IV) = C2 XOR (C1 XOR P1)
{code}
*2.*
*+1* for having two layer keys: EZ key and DEK :-). They are 3 points:
* It’s demonstrated in other traditional FS or Database, such as Oracle
transparent encryption. Fully agree with [~tucu00] "It’s a more secure
solution that EZ key can’t be accessed by client". Actually it’s also the
initial thought of two layer keys in HADOOP-10050,
* Truly support Key rotation, that’s very important. If only having EZ key,
even we have key versions, and for key rotation, we can use new version keys
for the new files, but for old files, they are still encrypted by the old
version keys (Unexpected user can still decrypt it). If we have two layer keys,
we can encrypt DEK keys using new EZ key version if necessary without decrypted
and encrypted the whole file.
* More easier management.
*3.*
{code}
KeyVersion generateEncryptedKey(String keyVersionName, byte[] iv)
KeyVersion decryptEncryptedKey(String keyVersionName, byte[] iv, KeyVersion
encryptedKey
{code}
We don’t need the iv? Parameter {{iv}} is for?
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: ConceptualDesignProposal_2014-06-19.pdf,
> HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: ConceptualDesignProposal_2014-06-19.pdf [~sanjay.radia], all, I'm attaching a 1 pager with the current Conceptual Design for HDFS encryption. On purpose I've left out details like IVs, key versions, key rotation support, etc. Later I'll put together a detailed Technical Design document. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: ConceptualDesignProposal_2014-06-19.pdf, > HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-5189) Rename the "CorruptBlocks" metric to "CorruptReplicas"
[ https://issues.apache.org/jira/browse/HDFS-5189?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Juan Yu reassigned HDFS-5189: - Assignee: Juan Yu > Rename the "CorruptBlocks" metric to "CorruptReplicas" > -- > > Key: HDFS-5189 > URL: https://issues.apache.org/jira/browse/HDFS-5189 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.1.0-beta >Reporter: Harsh J >Assignee: Juan Yu >Priority: Minor > > The NameNode increments a "CorruptBlocks" metric even if only one of the > block's > replicas is reported corrupt (genuine checksum fail, or even if a > replica has a bad genstamp). In cases where this is incremented, fsck > still reports a healthy state. > This is confusing to users and causes false alarm as they feel this is to be > monitored (instead of MissingBlocks). The metric is truly trying to report > only corrupt replicas, not whole blocks, and ought to be renamed. > FWIW, the "dfsadmin -report" reports a proper string of "Blocks with corrupt > replicas:" when printing this count. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038298#comment-14038298 ] Yongjun Zhang commented on HDFS-6403: - Many thanks ATM! > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.5.0 > > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6403: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the contribution, Yongjun. > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Fix For: 2.5.0 > > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038289#comment-14038289
]
Hadoop QA commented on HDFS-5546:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651573/HDFS-5546.2.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7184//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7184//console
This message is automatically generated.
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch, HDFS-5546.2.003.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[
https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038287#comment-14038287
]
Hadoop QA commented on HDFS-6403:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651546/HDFS-6403.005.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7183//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7183//console
This message is automatically generated.
> Add metrics for log warnings reported by JVM pauses
> ---
>
> Key: HDFS-6403
> URL: https://issues.apache.org/jira/browse/HDFS-6403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch,
> HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch
>
>
> HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed
> as a metric, then they can be monitored.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038249#comment-14038249
]
Hadoop QA commented on HDFS-6222:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651547/HDFS-6222.trunk-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7182//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7182//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7182//console
This message is automatically generated.
> Remove background token renewer from webhdfs
>
>
> Key: HDFS-6222
> URL: https://issues.apache.org/jira/browse/HDFS-6222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch,
> HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch,
> HDFS-6222.trunk.patch, HDFS-6222.trunk.patch
>
>
> The background token renewer is a source of problems for long-running
> daemons. Webhdfs should lazy fetch a new token when it receives an
> InvalidToken exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6507) Improve DFSAdmin to support HA cluster better
[ https://issues.apache.org/jira/browse/HDFS-6507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038235#comment-14038235 ] Zesheng Wu commented on HDFS-6507: -- Hi [~vinayrpet], it seems that folks do not have other opinions. Can you help to commit the patch? > Improve DFSAdmin to support HA cluster better > - > > Key: HDFS-6507 > URL: https://issues.apache.org/jira/browse/HDFS-6507 > Project: Hadoop HDFS > Issue Type: Improvement > Components: tools >Affects Versions: 2.4.0 >Reporter: Zesheng Wu >Assignee: Zesheng Wu > Attachments: HDFS-6507.1.patch, HDFS-6507.2.patch, HDFS-6507.3.patch, > HDFS-6507.4-inprogress.patch, HDFS-6507.4.patch, HDFS-6507.5.patch, > HDFS-6507.6.patch, HDFS-6507.7.patch, HDFS-6507.7.patch, HDFS-6507.8.patch > > > Currently, the commands supported in DFSAdmin can be classified into three > categories according to the protocol used: > 1. ClientProtocol > Commands in this category generally implement by calling the corresponding > function of the DFSClient class, and will call the corresponding remote > implementation function at the NN side finally. At the NN side, all these > operations are classified into five categories: UNCHECKED, READ, WRITE, > CHECKPOINT, JOURNAL. Active NN will allow all operations, and Standby NN only > allows UNCHECKED operations. In the current implementation of DFSClient, it > will connect one NN first, if the first NN is not Active and the operation is > not allowed, it will failover to the second NN. So here comes the problem, > some of the commands(setSafeMode, saveNameSpace, restoreFailedStorage, > refreshNodes, setBalancerBandwidth, metaSave) in DFSAdmin are classified as > UNCHECKED operations, and when executing these commands in the DFSAdmin > command line, they will be sent to a definite NN, no matter it is Active or > Standby. This may result in two problems: > a. If the first tried NN is standby, and the operation takes effect only on > Standby NN, which is not the expected result. > b. If the operation needs to take effect on both NN, but it takes effect on > only one NN. In the future, when there is a NN failover, there may have > problems. > Here I propose the following improvements: > a. If the command can be classified as one of READ/WRITE/CHECKPOINT/JOURNAL > operations, we should classify it clearly. > b. If the command can not be classified as one of the above four operations, > or if the command needs to take effect on both NN, we should send the request > to both Active and Standby NNs. > 2. Refresh protocols: RefreshAuthorizationPolicyProtocol, > RefreshUserMappingsProtocol, RefreshUserMappingsProtocol, > RefreshCallQueueProtocol > Commands in this category, including refreshServiceAcl, > refreshUserToGroupMapping, refreshSuperUserGroupsConfiguration and > refreshCallQueue, are implemented by creating a corresponding RPC proxy and > sending the request to remote NN. In the current implementation, these > requests will be sent to a definite NN, no matter it is Active or Standby. > Here I propose that we sent these requests to both NNs. > 3. ClientDatanodeProtocol > Commands in this category are handled correctly, no need to improve. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6562:
-
Attachment: HDFS-6562.003.patch
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch, HDFS-6562.003.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038212#comment-14038212
]
Hadoop QA commented on HDFS-6562:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651524/HDFS-6562.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7181//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7181//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7181//console
This message is automatically generated.
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[ https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6572 started by Charles Lamb. > Add an option to the NameNode that prints the software and on-disk image > versions > - > > Key: HDFS-6572 > URL: https://issues.apache.org/jira/browse/HDFS-6572 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HDFS-6572.001.patch > > > The HDFS namenode should have a startup option that prints the metadata > versions of both the software and the on-disk version. This will be useful > for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[ https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6572: --- Attachment: HDFS-6572.001.patch Rev 1 of the patch. > Add an option to the NameNode that prints the software and on-disk image > versions > - > > Key: HDFS-6572 > URL: https://issues.apache.org/jira/browse/HDFS-6572 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HDFS-6572.001.patch > > > The HDFS namenode should have a startup option that prints the metadata > versions of both the software and the on-disk version. This will be useful > for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6573) TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the unit test
[ https://issues.apache.org/jira/browse/HDFS-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinghui Wang updated HDFS-6573: --- Fix Version/s: (was: 2.4.0) (was: 2.3.0) (was: 2.2.0) 2.4.1 > TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the > unit test > > > Key: HDFS-6573 > URL: https://issues.apache.org/jira/browse/HDFS-6573 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0, 2.3.0, 2.4.0 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 2.4.1 > > Attachments: HDFS-6573.patch > > > o.a.h.hdfs.server.namenode.NameNodeRpcServer#blockReport calls a function in > for debug mode, which uses the StorageBlockReport data passed in by the test > TestDeadDatanode#testDeadDatanode. The report block list is an long array of > size 3 in the test, but the blockReport expects the format of long array of > size 5. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[ https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6572: --- Status: Patch Available (was: In Progress) > Add an option to the NameNode that prints the software and on-disk image > versions > - > > Key: HDFS-6572 > URL: https://issues.apache.org/jira/browse/HDFS-6572 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Reporter: Charles Lamb >Assignee: Charles Lamb >Priority: Minor > Attachments: HDFS-6572.001.patch > > > The HDFS namenode should have a startup option that prints the metadata > versions of both the software and the on-disk version. This will be useful > for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6418) Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
[
https://issues.apache.org/jira/browse/HDFS-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6418:
--
Attachment: h6418_20140619.patch
h6418_20140619.patch: adds back the previously deleted constants.
> Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
> ---
>
> Key: HDFS-6418
> URL: https://issues.apache.org/jira/browse/HDFS-6418
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Steve Loughran
>Assignee: Tsz Wo Nicholas Sze
> Attachments: h6418_20140619.patch
>
>
> Code i have that compiles against HADOOP 2.4 doesn't build against trunk as
> someone took away {{DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY}} -apparently in
> HDFS-6181.
> I know the name was obsolete, but anyone who has compiled code using that
> reference -rather than cutting and pasting in the string- is going to find
> their code doesn't work.
> More subtly: that will lead to a link exception trying to run that code on a
> 2.5+ cluster.
> This is a regression: the old names need to go back in, even if they refer to
> the new names and are marked as deprecated
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6570) add api that enables checking if a user has certain permissions on a file
[ https://issues.apache.org/jira/browse/HDFS-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038188#comment-14038188 ] Arpit Agarwal commented on HDFS-6570: - Thanks for the clarification Chris. > add api that enables checking if a user has certain permissions on a file > - > > Key: HDFS-6570 > URL: https://issues.apache.org/jira/browse/HDFS-6570 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Chris Nauroth > > For some of the authorization modes in Hive, the servers in Hive check if a > given user has permissions on a certain file or directory. For example, the > storage based authorization mode allows hive table metadata to be modified > only when the user has access to the corresponding table directory on hdfs. > There are likely to be such use cases outside of Hive as well. > HDFS does not provide an api for such checks. As a result, the logic to check > if a user has permissions on a directory gets replicated in Hive. This > results in duplicate logic and there introduces possibilities for > inconsistencies in the interpretation of the permission model. This becomes a > bigger problem with the complexity of ACL logic. > HDFS should provide an api that provides functionality that is similar to > access function in unistd.h - http://linux.die.net/man/2/access . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lei (Eddy) Xu updated HDFS-5546:
Attachment: HDFS-5546.2.003.patch
This patch absorts FNF exception but in the end, it prints out a warning
message to suggest users to re-run {{ls}}. The print format is something like
the following:
{quote}
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir0
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir1
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir2
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir3
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir4
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir5
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir6
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir7
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir8
drw-rw-rw- - test test 1024 1969-12-31 16:00 mockfs:///test/dir9
-rw-rw-rw- 1 test test 1024 1969-12-31 16:00 mockfs:///other/file
Warning: Files are deleted or renamed during running this command. Suggest to
re-run this command.
{quote}
Actually, all {{/test/dir#}} are deleted, but they are printed
anyway. The reason is that {{ls}} firstly prints out the current directory,
then jumps into the sub-directory recursively, there is no cheap way to test
the existence of the current directory before printing its information.
I think in this case, we should not directly catch IOExecption and
{{displayError(e)}}. {{LS}} command should be tolerate this scenario without
generating too much noise.
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch, HDFS-5546.2.003.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038165#comment-14038165 ] Tsz Wo Nicholas Sze commented on HDFS-4667: --- Patch looks good. Some comments: - Need to document how rename is captured in the diff report. - Add a getRenameEntry(inodeId) to SnapshotDiffInfo. - I suggest move SnapshotDiffInfo out from INodeDirectorySnapshottable since it is getting big. We should also refactor some other snapshot diff report code out. We may do it separately. - The patch does not apply any more. Need to update it. > Capture renamed files/directories in snapshot diff report > - > > Key: HDFS-4667 > URL: https://issues.apache.org/jira/browse/HDFS-4667 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, namenode >Reporter: Jing Zhao >Assignee: Binglin Chang > Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, > HDFS-4667.003.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, > getfullname-snapshot-support.patch > > > Currently in the diff report we only show file/dir creation, deletion and > modification. After rename with snapshots is supported, renamed file/dir > should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6573) TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the unit test
[ https://issues.apache.org/jira/browse/HDFS-6573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinghui Wang updated HDFS-6573: --- Attachment: HDFS-6573.patch > TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the > unit test > > > Key: HDFS-6573 > URL: https://issues.apache.org/jira/browse/HDFS-6573 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.2.0, 2.3.0, 2.4.0 >Reporter: Jinghui Wang >Assignee: Jinghui Wang >Priority: Minor > Fix For: 2.2.0, 2.3.0, 2.4.0 > > Attachments: HDFS-6573.patch > > > o.a.h.hdfs.server.namenode.NameNodeRpcServer#blockReport calls a function in > for debug mode, which uses the StorageBlockReport data passed in by the test > TestDeadDatanode#testDeadDatanode. The report block list is an long array of > size 3 in the test, but the blockReport expects the format of long array of > size 5. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Work started] (HDFS-6516) Implement List Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6516 started by Charles Lamb. > Implement List Encryption Zones > --- > > Key: HDFS-6516 > URL: https://issues.apache.org/jira/browse/HDFS-6516 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: security >Reporter: Charles Lamb >Assignee: Charles Lamb > > The list Encryption Zones command (CLI) and backend implementation > (FSNamesystem) needs to be implemented. As part of this, the tests in > TestEncryptionZonesAPI should be updated to use that to validate the results > of the various CreateEZ and DeleteEZ tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6573) TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the unit test
Jinghui Wang created HDFS-6573: -- Summary: TestDeadDatanode#testDeadDatanode fails if debug flag is turned on for the unit test Key: HDFS-6573 URL: https://issues.apache.org/jira/browse/HDFS-6573 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.4.0, 2.3.0, 2.2.0 Reporter: Jinghui Wang Assignee: Jinghui Wang Priority: Minor Fix For: 2.4.0, 2.3.0, 2.2.0 o.a.h.hdfs.server.namenode.NameNodeRpcServer#blockReport calls a function in for debug mode, which uses the StorageBlockReport data passed in by the test TestDeadDatanode#testDeadDatanode. The report block list is an long array of size 3 in the test, but the blockReport expects the format of long array of size 5. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038147#comment-14038147 ] Hudson commented on HDFS-6312: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5740 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5740/]) HDFS-6312. WebHdfs HA failover is broken on secure clusters. (daryn via tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604045) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java > WebHdfs HA failover is broken on secure clusters > > > Key: HDFS-6312 > URL: https://issues.apache.org/jira/browse/HDFS-6312 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6312.attempted.patch > > > When webhdfs does a failover, it blanks out the delegation token. This will > cause subsequent operations against the other NN to acquire a new token. > Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6570) add api that enables checking if a user has certain permissions on a file
[
https://issues.apache.org/jira/browse/HDFS-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038145#comment-14038145
]
Chris Nauroth commented on HDFS-6570:
-
Before submitting this issue, Thejas and I discussed trying to do this by
running existing {{FileSystem}} APIs inside a {{UserGroupInformation#doAs}}
block. Unfortunately, the permissions enforced by existing APIs do not match
exactly with the requirements of Hive. Also, this could have some unwanted
side effects, particularly for checking write access. This could unnecessarily
hold the write lock and write to the journal. Running an API like {{access}}
inside a {{UserGroupInformation#doAs}} would suit Hive's requirements better.
> add api that enables checking if a user has certain permissions on a file
> -
>
> Key: HDFS-6570
> URL: https://issues.apache.org/jira/browse/HDFS-6570
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Chris Nauroth
>
> For some of the authorization modes in Hive, the servers in Hive check if a
> given user has permissions on a certain file or directory. For example, the
> storage based authorization mode allows hive table metadata to be modified
> only when the user has access to the corresponding table directory on hdfs.
> There are likely to be such use cases outside of Hive as well.
> HDFS does not provide an api for such checks. As a result, the logic to check
> if a user has permissions on a directory gets replicated in Hive. This
> results in duplicate logic and there introduces possibilities for
> inconsistencies in the interpretation of the permission model. This becomes a
> bigger problem with the complexity of ACL logic.
> HDFS should provide an api that provides functionality that is similar to
> access function in unistd.h - http://linux.die.net/man/2/access .
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
Charles Lamb created HDFS-6572: -- Summary: Add an option to the NameNode that prints the software and on-disk image versions Key: HDFS-6572 URL: https://issues.apache.org/jira/browse/HDFS-6572 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor The HDFS namenode should have a startup option that prints the metadata versions of both the software and the on-disk version. This will be useful for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038125#comment-14038125 ] Aaron T. Myers commented on HDFS-6134: -- bq. This is not what one wants - distcp will not necessarily have permission in decrypt. I disagree - this is exactly what one wants. This is no different than today's distcp which may be run by a user that doesn't have permissions on all the files under the source directory. bq. delegation tokens - KMS will accept delegation tokens - again I don't think this is what one wants - can the keys be obtained at job submission time? Owen and Tucu have already discussed this quite a bit above. bq. I assume the NN will hand this out based on the file ACL. Does the above reduce the security? I don't see how this reduces security. The intention of adding transparent encryption support is just that - to provide encryption, not to provide another, additional authorization mechanism. bq. There are a complex set of issues to be addressed. I know that a bunch of you have had some private meetings discussing the various options and tradeoffs. Can we please have a short more public meeting next week? I can organize and host this at Hortonworks along with Google plus for those that are remote. How about next thursday at 1:30pm? I think those working on this project have been very open about all of these designs and discussions from the beginning dating back to March, and I think Tucu and others have been doing a great job of answering questions, accepting feedback, and modifying the design accordingly. Not sure where the assertion about private meetings is coming from - everything that's been discussed off-JIRA has been reiterated back on JIRA. What questions do you have remaining that would require a meeting? > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038106#comment-14038106 ] Sanjay Radia commented on HDFS-6134: There are a complex set of issues to be addressed. I know that a bunch of you have had some private meetings discussing the various options and tradeoffs. Can we please have a short more public meeting next week? I can organize and host this at Hortonworks along with Google plus for those that are remote. How about next thursday at 1:30pm? > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038099#comment-14038099 ] Sanjay Radia commented on HDFS-6134: * distcp and such tools and applications bq. Vanilla distcp will just work with transparent encryption. This is not what one wants - distcp will not necessarily have permission in decrypt. * delegation tokens - KMS will accept delegation tokens - again I don't think this is what one wants - can the keys be obtained at job submission time? * File ACLs bq. The NN gives the HDFS client the encrypted DEK and the keyVersion ID. I assume the NN will hand this out based on the file ACL. Does the above reduce the security? > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Assigned] (HDFS-6565) Use jackson instead jetty json in hdfs-client
[ https://issues.apache.org/jira/browse/HDFS-6565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Akira AJISAKA reassigned HDFS-6565: --- Assignee: Akira AJISAKA > Use jackson instead jetty json in hdfs-client > - > > Key: HDFS-6565 > URL: https://issues.apache.org/jira/browse/HDFS-6565 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Haohui Mai >Assignee: Akira AJISAKA > > hdfs-client should use Jackson instead of jetty to parse JSON. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5442) Zero loss HDFS data replication for multiple datacenters
[ https://issues.apache.org/jira/browse/HDFS-5442?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038091#comment-14038091 ] Chris Nauroth commented on HDFS-5442: - Thanks for the response, [~vinayrpet]. I read your comments and the revised design document, but I still have a few questions. bq. But in case of async mode, operations will not fail, but there will be a resync mechanism. bq. Since as mentioned above, there will not be chance of missing the edits, Operator need not discard complete instance of any DC. I didn't understand this response, because I didn't find anything in the design about a "resync mechanism" or something that can reconcile the two divergent namespaces, and of course this is quite a hard problem in general. This leads me to believe that after a split-brain accident (ouch!), the only possible recovery is to pick the namespace that is "better" according to the operator's judgment, discard the other one, bootstrap a new mirror, and wait for a full resync from the primary. Is that right, or am I missing something? Just to reiterate, I think claiming that we can switch to the mirror "in a matter of minutes" is over-stating the capabilities. Even for a quick admin that has addressed split-brain concerns and has the failover steps fully scripted, there is still the block reporting and safe mode time to consider. I'm sure some admins will want to fail back over to their "usual primary" even after activating the mirror. To do this, I imagine the admin will need to put the new primary into safe mode to block incoming writes, and then make sure the original primary DC gets caught up with the new primary on any asynchronous metadata or block writes. Will there be a tool to help admins check this? Without consideration of software upgrades, it looks like deployments that use this will face a tough choice between sticking to an old version vs. bootstrapping a whole new mirror cluster from scratch and doing a possibly costly full re-replication. > Zero loss HDFS data replication for multiple datacenters > > > Key: HDFS-5442 > URL: https://issues.apache.org/jira/browse/HDFS-5442 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Avik Dey >Assignee: Dian Fu > Attachments: Disaster Recovery Solution for Hadoop.pdf, Disaster > Recovery Solution for Hadoop.pdf, Disaster Recovery Solution for Hadoop.pdf > > > Hadoop is architected to operate efficiently at scale for normal hardware > failures within a datacenter. Hadoop is not designed today to handle > datacenter failures. Although HDFS is not designed for nor deployed in > configurations spanning multiple datacenters, replicating data from one > location to another is common practice for disaster recovery and global > service availability. There are current solutions available for batch > replication using data copy/export tools. However, while providing some > backup capability for HDFS data, they do not provide the capability to > recover all your HDFS data from a datacenter failure and be up and running > again with a fully operational Hadoop cluster in another datacenter in a > matter of minutes. For disaster recovery from a datacenter failure, we should > provide a fully distributed, zero data loss, low latency, high throughput and > secure HDFS data replication solution for multiple datacenter setup. > Design and code for Phase-1 to follow soon. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038084#comment-14038084 ] Yongjun Zhang commented on HDFS-6312: - Many thanks Alejandro and Daryn! > WebHdfs HA failover is broken on secure clusters > > > Key: HDFS-6312 > URL: https://issues.apache.org/jira/browse/HDFS-6312 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6312.attempted.patch > > > When webhdfs does a failover, it blanks out the delegation token. This will > cause subsequent operations against the other NN to acquire a new token. > Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6570) add api that enables checking if a user has certain permissions on a file
[ https://issues.apache.org/jira/browse/HDFS-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038077#comment-14038077 ] Arpit Agarwal commented on HDFS-6570: - Is it possible to impersonate the user and try to open the file with the permissions you are interested in? If it succeeds the user has the permissions. > add api that enables checking if a user has certain permissions on a file > - > > Key: HDFS-6570 > URL: https://issues.apache.org/jira/browse/HDFS-6570 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: Thejas M Nair >Assignee: Chris Nauroth > > For some of the authorization modes in Hive, the servers in Hive check if a > given user has permissions on a certain file or directory. For example, the > storage based authorization mode allows hive table metadata to be modified > only when the user has access to the corresponding table directory on hdfs. > There are likely to be such use cases outside of Hive as well. > HDFS does not provide an api for such checks. As a result, the logic to check > if a user has permissions on a directory gets replicated in Hive. This > results in duplicate logic and there introduces possibilities for > inconsistencies in the interpretation of the permission model. This becomes a > bigger problem with the complexity of ACL logic. > HDFS should provide an api that provides functionality that is similar to > access function in unistd.h - http://linux.die.net/man/2/access . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6312: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks Daryn and Yongjun. Committed to trunk and branch-2. (on the commit message i've given credit to Daryn as i saw the JIRA was assigned to him, sorry for the mishap). > WebHdfs HA failover is broken on secure clusters > > > Key: HDFS-6312 > URL: https://issues.apache.org/jira/browse/HDFS-6312 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 3.0.0, 2.4.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Fix For: 2.5.0 > > Attachments: HDFS-6312.attempted.patch > > > When webhdfs does a failover, it blanks out the delegation token. This will > cause subsequent operations against the other NN to acquire a new token. > Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038065#comment-14038065 ] Aaron T. Myers commented on HDFS-6403: -- Great, thanks. +1 pending Jenkins. > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Status: Open (was: Patch Available) Uploaded patches for branch-2 and trunk. Jenkins took the branch-2 patch and commented that it could not apply the patch against trunk revision. So cancelling the patch. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Status: Patch Available (was: Open) > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Attachment: HDFS-6222.trunk-v2.patch Uploading the trunk patch only. > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, > HDFS-6222.trunk.patch, HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[
https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038058#comment-14038058
]
Hadoop QA commented on HDFS-6312:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12650623/HDFS-6312.attempted.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7179//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7179//console
This message is automatically generated.
> WebHdfs HA failover is broken on secure clusters
>
>
> Key: HDFS-6312
> URL: https://issues.apache.org/jira/browse/HDFS-6312
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 3.0.0, 2.4.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Blocker
> Attachments: HDFS-6312.attempted.patch
>
>
> When webhdfs does a failover, it blanks out the delegation token. This will
> cause subsequent operations against the other NN to acquire a new token.
> Tasks cannot acquire a token (no kerberos credentials) so jobs will fail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038057#comment-14038057 ] Yongjun Zhang commented on HDFS-6403: - Thanks ATM, good catch! removed it and updated new version. > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yongjun Zhang updated HDFS-6403: Attachment: HDFS-6403.005.patch > Add metrics for log warnings reported by JVM pauses > --- > > Key: HDFS-6403 > URL: https://issues.apache.org/jira/browse/HDFS-6403 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode, namenode >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, > HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch > > > HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed > as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038040#comment-14038040
]
Aaron T. Myers commented on HDFS-6475:
--
+1, the latest patch looks good to me.
I'll commit this tomorrow unless anyone else has comments in the meantime.
> WebHdfs clients fail without retry because incorrect handling of
> StandbyException
> -
>
> Key: HDFS-6475
> URL: https://issues.apache.org/jira/browse/HDFS-6475
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, webhdfs
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
> HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
> HDFS-6475.005.patch, HDFS-6475.006.patch
>
>
> With WebHdfs clients connected to a HA HDFS service, the delegation token is
> previously initialized with the active NN.
> When clients try to issue request, the NN it contacts is stored in a map
> returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
> the NN based on the order, so likely the first one it runs into is StandbyNN.
> If the StandbyNN doesn't have the updated client crediential, it will throw a
> s SecurityException that wraps StandbyException.
> The client is expected to retry another NN, but due to the insufficient
> handling of SecurityException mentioned above, it failed.
> Example message:
> {code}
> {RemoteException={message=Failed to obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken:
> StandbyException, javaCl
> assName=java.lang.SecurityException, exception=SecurityException}}
> org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
> obtain user group information:
> org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
> at
> org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
> at
> org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
> at kclient1.kclient$1.run(kclient.java:64)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:356)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
> at kclient1.kclient.main(kclient.java:58)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
> {code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6571) NameNode should delete intermediate fsimage.ckpt when checkpoint fails
Akira AJISAKA created HDFS-6571: --- Summary: NameNode should delete intermediate fsimage.ckpt when checkpoint fails Key: HDFS-6571 URL: https://issues.apache.org/jira/browse/HDFS-6571 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.0 Reporter: Akira AJISAKA When checkpoint fails in getting a new fsimage from standby NameNode or SecondaryNameNode, intermediate fsimage (fsimage.ckpt_) is left and never to be cleaned up. If fsimage is large and fails to checkpoint many times, the growing intermediate fsimage may cause out of disk space. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038014#comment-14038014
]
Hadoop QA commented on HDFS-6222:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651519/HDFS-6222.branch-2-v2.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7180//console
This message is automatically generated.
> Remove background token renewer from webhdfs
>
>
> Key: HDFS-6222
> URL: https://issues.apache.org/jira/browse/HDFS-6222
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: webhdfs
>Affects Versions: 2.0.0-alpha, 3.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
> Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch,
> HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk.patch,
> HDFS-6222.trunk.patch
>
>
> The background token renewer is a source of problems for long-running
> daemons. Webhdfs should lazy fetch a new token when it receives an
> InvalidToken exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[
https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14038010#comment-14038010
]
Aaron T. Myers commented on HDFS-6403:
--
Hi Yongjun, the latest patch looks pretty good to me. Thanks a lot for
incorporation my feedback.
One small comment: I think that the new method {{JvmMetrics#getPauseMonitor}}
is currently unused. Mind removing that?
+1 once this is addressed.
> Add metrics for log warnings reported by JVM pauses
> ---
>
> Key: HDFS-6403
> URL: https://issues.apache.org/jira/browse/HDFS-6403
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode, namenode
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
> Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch,
> HDFS-6403.003.patch, HDFS-6403.004.patch
>
>
> HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed
> as a metric, then they can be monitored.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037999#comment-14037999
]
Lei (Eddy) Xu commented on HDFS-5546:
-
It appears to me that the actual behavior is that the FNF exceptions are
printed by {{Command#displayError(e)}} in the middle of LS output, which looks
like that it terminates `LS` because of FNF. I will write a test case to verify
this.
I am actually working on a change that only prints out existing
directories/files and puts the captured FNF exception to
{{Command#exceptions}}. In the end of {{LS}}, it checks whether there is FNF in
{{exceptions}} and prints a human readable message if there is one or more FNF.
What do you think, [~daryn] and [~cmccabe]?
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6562:
-
Attachment: HDFS-6562.002.patch
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
> HDFS-6562.002.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Status: Patch Available (was: Open) Added a bunch of assert statements for token kind and added a test case for SWebHdfsFileSystem > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk.patch, > HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6562:
-
Attachment: HDFS-6562.001.patch
> Refactor rename() in FSDirectory
>
>
> Key: HDFS-6562
> URL: https://issues.apache.org/jira/browse/HDFS-6562
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch
>
>
> Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
> Both implementation shares quite a bit of common code.
> This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Status: Open (was: Patch Available) On behalf of daryn: Cancelling this patch in order to incorporate Chris's comments > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk.patch, > HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Attachment: HDFS-6222.branch-2-v2.patch HDFS-6222.trunk-v2.patch > Remove background token renewer from webhdfs > > > Key: HDFS-6222 > URL: https://issues.apache.org/jira/browse/HDFS-6222 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, > HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk.patch, > HDFS-6222.trunk.patch > > > The background token renewer is a source of problems for long-running > daemons. Webhdfs should lazy fetch a new token when it receives an > InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-5546) race condition crashes "hadoop ls -R" when directories are moved/removed
[
https://issues.apache.org/jira/browse/HDFS-5546?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037978#comment-14037978
]
Colin Patrick McCabe commented on HDFS-5546:
Hmm. I think you're right about this. {{Command#processPaths}} wraps the call
to {{recursePath}} in an IOE try... catch block. So if we can't recurse into
an individual child, we should still be able to move forward with the other
ones. Of course, there is no such protection for the paths specified on the
command-line. I tried looking for all the places IOE could be thrown, but
didn't manage to spot one that would abort the recursion because of a problem
with a child. Eddy, can you run your unit test against trunk to verify all
this?
> race condition crashes "hadoop ls -R" when directories are moved/removed
>
>
> Key: HDFS-5546
> URL: https://issues.apache.org/jira/browse/HDFS-5546
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.2.0
>Reporter: Colin Patrick McCabe
>Assignee: Lei (Eddy) Xu
>Priority: Minor
> Fix For: 3.0.0
>
> Attachments: HDFS-5546.1.patch, HDFS-5546.2.000.patch,
> HDFS-5546.2.001.patch, HDFS-5546.2.002.patch
>
>
> This seems to be a rare race condition where we have a sequence of events
> like this:
> 1. org.apache.hadoop.shell.Ls calls DFS#getFileStatus on directory D.
> 2. someone deletes or moves directory D
> 3. org.apache.hadoop.shell.Ls calls PathData#getDirectoryContents(D), which
> calls DFS#listStatus(D). This throws FileNotFoundException.
> 4. ls command terminates with FNF
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6418) Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
[
https://issues.apache.org/jira/browse/HDFS-6418?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037946#comment-14037946
]
Tsz Wo Nicholas Sze commented on HDFS-6418:
---
I agree with Suresh and Jing that DFSConfigKeys is not a public API so that it
should not be used outside HDFS. On the other hand, the conf property key is a
public API. For example,
{code}
//DFSConfigKeys.java
public static final String DFS_BLOCK_SIZE_KEY = "dfs.blocksize";
{code}
DFS_BLOCK_SIZE_KEY is not a public API. Projects outside HDFS should not use
DFSConfigKeys.DFS_BLOCK_SIZE_KEY. However, the value "dfs.blocksize" is a
public API as it is defined in hdfs-default.xml.
In the long term, Slider should eliminate referencing DFSConfigKeys. In the
short term, I don't mind adding back the previously removed constants as
deprecated. Sounds good?
> Regression: DFS_NAMENODE_USER_NAME_KEY missing in trunk
> ---
>
> Key: HDFS-6418
> URL: https://issues.apache.org/jira/browse/HDFS-6418
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client
>Affects Versions: 3.0.0, 2.5.0
>Reporter: Steve Loughran
>Assignee: Tsz Wo Nicholas Sze
>
> Code i have that compiles against HADOOP 2.4 doesn't build against trunk as
> someone took away {{DFSConfigKeys.DFS_NAMENODE_USER_NAME_KEY}} -apparently in
> HDFS-6181.
> I know the name was obsolete, but anyone who has compiled code using that
> reference -rather than cutting and pasting in the string- is going to find
> their code doesn't work.
> More subtly: that will lead to a link exception trying to run that code on a
> 2.5+ cluster.
> This is a regression: the old names need to go back in, even if they refer to
> the new names and are marked as deprecated
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6570) add api that enables checking if a user has certain permissions on a file
[
https://issues.apache.org/jira/browse/HDFS-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth reassigned HDFS-6570:
---
Assignee: Chris Nauroth
I'm tentatively assigning this to myself, since [~thejas] and I have had some
discussion that prompted filing this issue.
Note that the man page for {{access}} clearly spells out the risk of
time-of-check/time-of-use race conditions. This API is only going to be useful
for systems implementing their own authorization enforcement on top of HDFS
files, and only if those systems consider the risk acceptable.
> add api that enables checking if a user has certain permissions on a file
> -
>
> Key: HDFS-6570
> URL: https://issues.apache.org/jira/browse/HDFS-6570
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Thejas M Nair
>Assignee: Chris Nauroth
>
> For some of the authorization modes in Hive, the servers in Hive check if a
> given user has permissions on a certain file or directory. For example, the
> storage based authorization mode allows hive table metadata to be modified
> only when the user has access to the corresponding table directory on hdfs.
> There are likely to be such use cases outside of Hive as well.
> HDFS does not provide an api for such checks. As a result, the logic to check
> if a user has permissions on a directory gets replicated in Hive. This
> results in duplicate logic and there introduces possibilities for
> inconsistencies in the interpretation of the permission model. This becomes a
> bigger problem with the complexity of ACL logic.
> HDFS should provide an api that provides functionality that is similar to
> access function in unistd.h - http://linux.die.net/man/2/access .
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6570) add api that enables checking if a user has certain permissions on a file
Thejas M Nair created HDFS-6570: --- Summary: add api that enables checking if a user has certain permissions on a file Key: HDFS-6570 URL: https://issues.apache.org/jira/browse/HDFS-6570 Project: Hadoop HDFS Issue Type: Bug Reporter: Thejas M Nair For some of the authorization modes in Hive, the servers in Hive check if a given user has permissions on a certain file or directory. For example, the storage based authorization mode allows hive table metadata to be modified only when the user has access to the corresponding table directory on hdfs. There are likely to be such use cases outside of Hive as well. HDFS does not provide an api for such checks. As a result, the logic to check if a user has permissions on a directory gets replicated in Hive. This results in duplicate logic and there introduces possibilities for inconsistencies in the interpretation of the permission model. This becomes a bigger problem with the complexity of ACL logic. HDFS should provide an api that provides functionality that is similar to access function in unistd.h - http://linux.die.net/man/2/access . -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037869#comment-14037869
]
Hadoop QA commented on HDFS-6557:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651460/HDFS-6557.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7177//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7177//console
This message is automatically generated.
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6557.000.patch, HDFS-6557.001.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-5477) Block manager as a service
[ https://issues.apache.org/jira/browse/HDFS-5477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037858#comment-14037858 ] Edward Bortnikov commented on HDFS-5477: The published document supersedes all the previous posts > Block manager as a service > -- > > Key: HDFS-5477 > URL: https://issues.apache.org/jira/browse/HDFS-5477 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.0.0-alpha, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp > Attachments: Block-Manager-as-a-Service.pdf, Proposal.pdf, > Proposal.pdf, Standalone BM.pdf, Standalone BM.pdf > > > The block manager needs to evolve towards having the ability to run as a > standalone service to improve NN vertical and horizontal scalability. The > goal is reducing the memory footprint of the NN proper to support larger > namespaces, and improve overall performance by decoupling the block manager > from the namespace and its lock. Ideally, a distinct BM will be transparent > to clients and DNs. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HDFS-6391. Resolution: Not a Problem subsumed by HDFS-6474 > Get the Key/IV from the NameNode for encrypted files in DFSClient > - > > Key: HDFS-6391 > URL: https://issues.apache.org/jira/browse/HDFS-6391 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Alejandro Abdelnur >Assignee: Andrew Wang > Attachments: HDFS-6391.1.patch > > > When creating/opening and encrypted file, the DFSClient should get the > encryption key material and the IV for the file in the create/open RPC call. > HDFS admin users would never get key material/IV on encrypted files > create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037814#comment-14037814 ] Brandon Li commented on HDFS-6569: -- Here is the steps to reproduce it: 1. upload 1GB file using "hadoop dfs -put" 2. during the file loading, do a "hadoop dfsadmin -shutdownDatanode host:port upgrade". This will cause the write failure. > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037812#comment-14037812 ] Brandon Li commented on HDFS-6569: -- Yes. It's a single node cluster with replication factor as 1. > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6312: - Status: Patch Available (was: Open) > WebHdfs HA failover is broken on secure clusters > > > Key: HDFS-6312 > URL: https://issues.apache.org/jira/browse/HDFS-6312 > Project: Hadoop HDFS > Issue Type: Bug > Components: webhdfs >Affects Versions: 2.4.0, 3.0.0 >Reporter: Daryn Sharp >Assignee: Daryn Sharp >Priority: Blocker > Attachments: HDFS-6312.attempted.patch > > > When webhdfs does a failover, it blanks out the delegation token. This will > cause subsequent operations against the other NN to acquire a new token. > Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6490) Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone
[
https://issues.apache.org/jira/browse/HDFS-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charles Lamb updated HDFS-6490:
---
Attachment: HDFS-6490.001.patch
Attached is a patch which addresses this. Generated keys are formatted as:
hdfs:{UUID}
If DFSUtil.getNamenodeNameServiceId() returns null, then the is
blank.
For example, the keyId for path "/src"
hdfs:///src/e4723785-c4f2-48bd-892e-4d80ef4d5af7
I don't find this format especially appealing and I'm happy to get other
suggestions.
> Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone
> -
>
> Key: HDFS-6490
> URL: https://issues.apache.org/jira/browse/HDFS-6490
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, security
>Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134)
>Reporter: Charles Lamb
>Assignee: Charles Lamb
> Attachments: HDFS-6490.001.patch
>
>
> FSNamesystem.createEncryptionZone needs to create key ids with the format
> hdfs://HOST:PORT/
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037778#comment-14037778 ] Alejandro Abdelnur commented on HDFS-6134: -- [~sanjay.radia], thanks for jumping in. Things has changed a bit since the latest design doc based on the received feedback, mostly from [~owen.omalley] and [~yoderme]. (I will update the design doc to reflect these changes). On *distcp:* Vanilla distcp will just work with transparent encryption. Data will be decrypted on read and encrypted on write, assuming both source and target are in encrypted zones. The proposal on changing distcp is to enable a second use used case, copy data from one cluster to another without having to decrypt/encrypt the data while doing the copy. This is useful when doing copies for disaster recovery, hdfs admins could do the copy without having to have access to the encryption keys. On *relying on kerberos credentials OR delegation tokens to obtain keys:* It works exactly like HDFS. KMS will support both Kerberos and delegation tokens. A Kerberized client can request a KMS delegation token which is serialized with the rest of the credentials to be used by containers running in the cluster. It is assumed you are using network encryption as well to avoid delegation tokens sniffing. On *relying on file-ACLs to hand out keys:* No, file-ACLs give you access to the data in HDFS. You also need to have access to the Key, that is the responsibility of the KeyProvider to do. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6531) Create a native api to delete file, like the renameTo method in NativeIO, for better error reporting
[ https://issues.apache.org/jira/browse/HDFS-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037782#comment-14037782 ] Yongjun Zhang commented on HDFS-6531: - Thanks for the linking and sharing, [~cnauroth]. Certainly a more systematic approach with JDK7 in HADOOP-9590. Haven't got to work on this one yet, good to know the other thread. > Create a native api to delete file, like the renameTo method in NativeIO, for > better error reporting > > > Key: HDFS-6531 > URL: https://issues.apache.org/jira/browse/HDFS-6531 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.4.0 >Reporter: Yongjun Zhang >Assignee: Yongjun Zhang > > File.delete() returns boolean to indicate success or failure. > As a follow-up of HDFS-6499, filing this jira to provide a native API to > delete file, like the renameTo method in NativeIO. So caller can tell better > what's the reason of failure. > And fix FileJournalManager etc places for better error reporting. > Thanks [~atm] for reviewing the fix of HDFS-6499 and pointing out this issue. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037776#comment-14037776
]
Andrew Purtell commented on HDFS-6134:
--
{quote}
A known solution to address this is:
A. Each file should use a unique data encryption key (DEK).
{quote}
FWIW, this is what HBase transparent encryption does, unless the admin is
explicitly providing their own DEKs.
> Transparent data at rest encryption
> ---
>
> Key: HDFS-6134
> URL: https://issues.apache.org/jira/browse/HDFS-6134
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: security
>Affects Versions: 2.3.0
>Reporter: Alejandro Abdelnur
>Assignee: Alejandro Abdelnur
> Attachments: HDFSDataAtRestEncryption.pdf
>
>
> Because of privacy and security regulations, for many industries, sensitive
> data at rest must be in encrypted form. For example: the healthcare industry
> (HIPAA regulations), the card payment industry (PCI DSS regulations) or the
> US government (FISMA regulations).
> This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
> be used transparently by any application accessing HDFS via Hadoop Filesystem
> Java API, Hadoop libhdfs C library, or WebHDFS REST API.
> The resulting implementation should be able to be used in compliance with
> different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Assigned] (HDFS-6474) Namenode needs to get the actual keys and iv from the KeyProvider
[ https://issues.apache.org/jira/browse/HDFS-6474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb reassigned HDFS-6474: -- Assignee: Andrew Wang (was: Charles Lamb) > Namenode needs to get the actual keys and iv from the KeyProvider > - > > Key: HDFS-6474 > URL: https://issues.apache.org/jira/browse/HDFS-6474 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Reporter: Charles Lamb >Assignee: Andrew Wang > > The Namenode has code to connect to the KeyProvider, but it needs to actually > get the keys and return them to the client at the right time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037752#comment-14037752 ] Hudson commented on HDFS-6549: -- FAILURE: Integrated in Hadoop-trunk-Commit #5738 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5738/]) HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS client. Contributed by Aaron T. Myers. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604022) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.5.0 > > Attachments: HDFS-6549.patch, HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6549: - Resolution: Fixed Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the reviews, Brandon and Juan. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.5.0 > > Attachments: HDFS-6549.patch, HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037737#comment-14037737 ] Sanjay Radia commented on HDFS-6134: bq. On the distcp not accessing the keys (not decrypting/encrypting), yes, that is the idea. Alejandro not sure if I understand what you mean by the above. Are you saying that distcp and other tools/applications that copy and backup data will have be changed to do something different when the file is encrypted? In a sense, this Jira's attempt to provide transparent encryption, is breaking existing transparency. Two other questions: * Are you relying in the the kerberos credentials OR delegation tokens to obtain the keys? Isn't using the delegation token to obtain keys reducing security? * Looks like the proposal relies on file-ACLs to hand out keys - part of the motivation for using encryption is that ACLs are often correctly set. > Transparent data at rest encryption > --- > > Key: HDFS-6134 > URL: https://issues.apache.org/jira/browse/HDFS-6134 > Project: Hadoop HDFS > Issue Type: New Feature > Components: security >Affects Versions: 2.3.0 >Reporter: Alejandro Abdelnur >Assignee: Alejandro Abdelnur > Attachments: HDFSDataAtRestEncryption.pdf > > > Because of privacy and security regulations, for many industries, sensitive > data at rest must be in encrypted form. For example: the healthcare industry > (HIPAA regulations), the card payment industry (PCI DSS regulations) or the > US government (FISMA regulations). > This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can > be used transparently by any application accessing HDFS via Hadoop Filesystem > Java API, Hadoop libhdfs C library, or WebHDFS REST API. > The resulting implementation should be able to be used in compliance with > different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6531) Create a native api to delete file, like the renameTo method in NativeIO, for better error reporting
[
https://issues.apache.org/jira/browse/HDFS-6531?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037732#comment-14037732
]
Chris Nauroth commented on HDFS-6531:
-
Linking to HADOOP-9590, which tracks our use cases for calling the new file
APIs introduced in JDK7. With JDK7, we could potentially use {{Files#delete}},
which throws an exception with details if the delete fails. This would reduce
our native code footprint.
http://docs.oracle.com/javase/7/docs/api/java/nio/file/Files.html#delete(java.nio.file.Path)
> Create a native api to delete file, like the renameTo method in NativeIO, for
> better error reporting
>
>
> Key: HDFS-6531
> URL: https://issues.apache.org/jira/browse/HDFS-6531
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.4.0
>Reporter: Yongjun Zhang
>Assignee: Yongjun Zhang
>
> File.delete() returns boolean to indicate success or failure.
> As a follow-up of HDFS-6499, filing this jira to provide a native API to
> delete file, like the renameTo method in NativeIO. So caller can tell better
> what's the reason of failure.
> And fix FileJournalManager etc places for better error reporting.
> Thanks [~atm] for reviewing the fix of HDFS-6499 and pointing out this issue.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6549: - Attachment: HDFS-6549.patch Sure is! Good catch, Juan. Attached patch fixes that typo. I'm going to go ahead and commit this based on Brandon's +1 without waiting for test-patch since the difference is only a docs typo. > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch, HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037715#comment-14037715 ] Hudson commented on HDFS-3848: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5737 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5737/]) HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class. Contributed by Hooman Peiro Sajjad and Chen He. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1604011) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery2.java > A Bug in recoverLeaseInternal method of FSNameSystem class > -- > > Key: HDFS-3848 > URL: https://issues.apache.org/jira/browse/HDFS-3848 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.1 >Reporter: Hooman Peiro Sajjad >Assignee: Chen He > Labels: patch > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This is a bug in logic of the method recoverLeaseInternal. In line 1322 it > checks if the owner of the file is trying to recreate the file. The condition > of the if statement is > (leaseFile != null && leaseFile.equals(lease)) || > lease.getHolder().equals(holder) > As it can be seen, there are two operands (conditions) connected with an "or" > operator. The first operand is straight and will be true only if the holder > of the file is the new holder. But the problem is the second operand which > will be always true since the "lease" object is the one found by the "holder" > by calling "Lease lease = leaseManager.getLease(holder);" in line 1315. > To fix this I think the if statement only should contain the following the > condition: > (leaseFile != null && leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-3848: - Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk and branch-2. Thanks for working on this Hooman and Chen. > A Bug in recoverLeaseInternal method of FSNameSystem class > -- > > Key: HDFS-3848 > URL: https://issues.apache.org/jira/browse/HDFS-3848 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.1 >Reporter: Hooman Peiro Sajjad >Assignee: Chen He > Labels: patch > Fix For: 3.0.0, 2.5.0 > > Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This is a bug in logic of the method recoverLeaseInternal. In line 1322 it > checks if the owner of the file is trying to recreate the file. The condition > of the if statement is > (leaseFile != null && leaseFile.equals(lease)) || > lease.getHolder().equals(holder) > As it can be seen, there are two operands (conditions) connected with an "or" > operator. The first operand is straight and will be true only if the holder > of the file is the new holder. But the problem is the second operand which > will be always true since the "lease" object is the one found by the "holder" > by calling "Lease lease = leaseManager.getLease(holder);" in line 1315. > To fix this I think the if statement only should contain the following the > condition: > (leaseFile != null && leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6569: - Affects Version/s: (was: 2.2.0) 2.4.0 > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037674#comment-14037674 ] Kihwal Lee commented on HDFS-6569: -- It is unclear who closed the connection. During the development, I fixed one multiple closure bug and exceptions caused by extra ones in datanode. There may be other cases remaining. Is it from a write using one replica? > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-3848: - Assignee: Chen He > A Bug in recoverLeaseInternal method of FSNameSystem class > -- > > Key: HDFS-3848 > URL: https://issues.apache.org/jira/browse/HDFS-3848 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.1 >Reporter: Hooman Peiro Sajjad >Assignee: Chen He > Labels: patch > Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This is a bug in logic of the method recoverLeaseInternal. In line 1322 it > checks if the owner of the file is trying to recreate the file. The condition > of the if statement is > (leaseFile != null && leaseFile.equals(lease)) || > lease.getHolder().equals(holder) > As it can be seen, there are two operands (conditions) connected with an "or" > operator. The first operand is straight and will be true only if the holder > of the file is the new holder. But the problem is the second operand which > will be always true since the "lease" object is the one found by the "holder" > by calling "Lease lease = leaseManager.getLease(holder);" in line 1315. > To fix this I think the if statement only should contain the following the > condition: > (leaseFile != null && leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037682#comment-14037682 ] Kihwal Lee commented on HDFS-3848: -- +1 the patch looks good. Hooman has been inactive for about two years. I will assign it to you Chen, but will give credit to all. > A Bug in recoverLeaseInternal method of FSNameSystem class > -- > > Key: HDFS-3848 > URL: https://issues.apache.org/jira/browse/HDFS-3848 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.1 >Reporter: Hooman Peiro Sajjad > Labels: patch > Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This is a bug in logic of the method recoverLeaseInternal. In line 1322 it > checks if the owner of the file is trying to recreate the file. The condition > of the if statement is > (leaseFile != null && leaseFile.equals(lease)) || > lease.getHolder().equals(holder) > As it can be seen, there are two operands (conditions) connected with an "or" > operator. The first operand is straight and will be true only if the holder > of the file is the new holder. But the problem is the second operand which > will be always true since the "lease" object is the one found by the "holder" > by calling "Lease lease = leaseManager.getLease(holder);" in line 1315. > To fix this I think the if statement only should contain the following the > condition: > (leaseFile != null && leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037678#comment-14037678 ] Chen He commented on HDFS-3848: --- Hi [~rike], If you are not working on this issue, I will take it. > A Bug in recoverLeaseInternal method of FSNameSystem class > -- > > Key: HDFS-3848 > URL: https://issues.apache.org/jira/browse/HDFS-3848 > Project: Hadoop HDFS > Issue Type: Bug > Components: namenode >Affects Versions: 0.23.1 >Reporter: Hooman Peiro Sajjad > Labels: patch > Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch > > Original Estimate: 1h > Remaining Estimate: 1h > > This is a bug in logic of the method recoverLeaseInternal. In line 1322 it > checks if the owner of the file is trying to recreate the file. The condition > of the if statement is > (leaseFile != null && leaseFile.equals(lease)) || > lease.getHolder().equals(holder) > As it can be seen, there are two operands (conditions) connected with an "or" > operator. The first operand is straight and will be true only if the holder > of the file is the new holder. But the problem is the second operand which > will be always true since the "lease" object is the one found by the "holder" > by calling "Lease lease = leaseManager.getLease(holder);" in line 1315. > To fix this I think the if statement only should contain the following the > condition: > (leaseFile != null && leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6561) Byte array native checksumming on client side
[ https://issues.apache.org/jira/browse/HDFS-6561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037675#comment-14037675 ] James Thomas commented on HDFS-6561: It seems like the best approach here would be to add some buffering to FSOutputSummer. I did some testing and it seems like going to native code is strictly faster than using the incremental summer as long as flushes happen no less often than every 100 bytes. More specifically, if we use the native code, which has no option for incremental checksumming (it could be added in, but I don't think it's necessary), if we compute a checksum of a partial chunk, we will need to recompute the entire checksum (rather than just the checksum of the newly added bytes) when the chunk either 1) fills out or 2) is incremented a bit and then a flush occurs. But the native checksum on 512 bytes (the chunk size and the maximum possible size of a partial chunk) takes at most 7 microseconds on my machine, and the incremental Java checksum requires around 7 microseconds for 100 bytes. So as long as it's reasonable to assume flushes won't happen more often than every 100 bytes, I propose that we eliminate the incremental summer from FSOutputSummer and always use the native code for byte arrays. Buffering FSOutputSummer will mean that writes on the wire are more bursty and data is kept in client memory longer, but Todd mentioned that many clients already wrap DFSOutputStream in a BufferedWriter anyway. To achieve maximum performance in the native code, we need the buffer to be at least a few thousand bytes. The larger the buffer, the better, since we can amortize away the cost of crossing the JNI boundary (which seems to be quite small actually, on the order of a microsecond at most). What do people think about this? > Byte array native checksumming on client side > - > > Key: HDFS-6561 > URL: https://issues.apache.org/jira/browse/HDFS-6561 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, hdfs-client, performance >Reporter: James Thomas >Assignee: James Thomas > -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Kihwal Lee updated HDFS-6569: - Target Version/s: 2.5.0 > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.4.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6569) OOB message can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-6569: - Summary: OOB message can't be sent to the client when DataNode shuts down for upgrade (was: OOB massage can't be sent to the client when DataNode shuts down for upgrade) > OOB message can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6569) OOB massage can't be sent to the client when DataNode shuts down for upgrade
[
https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037640#comment-14037640
]
Brandon Li commented on HDFS-6569:
--
The client stack trace:
{noformat}
14/06/19 11:26:23 WARN hdfs.DFSClient: DFSOutputStream ResponseProcessor
exception for block BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004
java.io.EOFException: Premature EOF: no length prefix available
at
org.apache.hadoop.hdfs.protocolPB.PBHelper.vintPrefixed(PBHelper.java:1997)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PipelineAck.readFields(PipelineAck.java:176)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer$ResponseProcessor.run(DFSOutputStream.java:793)
14/06/19 11:26:23 WARN hdfs.DFSClient: DataStreamer Exception
java.io.IOException: Broken pipe
at sun.nio.ch.FileDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:29)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:69)
at sun.nio.ch.IOUtil.write(IOUtil.java:40)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:336)
at
org.apache.hadoop.net.SocketOutputStream$Writer.performIO(SocketOutputStream.java:63)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:159)
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:117)
at java.io.BufferedOutputStream.write(BufferedOutputStream.java:105)
at java.io.DataOutputStream.write(DataOutputStream.java:90)
at
org.apache.hadoop.hdfs.DFSOutputStream$Packet.writeTo(DFSOutputStream.java:271)
at
org.apache.hadoop.hdfs.DFSOutputStream$DataStreamer.run(DFSOutputStream.java:574)
put: All datanodes 127.0.0.1:50010 are bad. Aborting...
{noformat}
The DataNode error:
{noformat}
2014-06-19 11:26:23,668 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Shutting down for restart
(BP-1063118919-10.11.1.167-1403201799003:blk_1073741828_1004).
2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Sending an out of band ack of type OOB_RESTART
2014-06-19 11:26:23,669 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
Error sending OOB Ack.
java.io.IOException: The stream is closed
at
org.apache.hadoop.net.SocketOutputStream.write(SocketOutputStream.java:118)
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
at java.io.DataOutputStream.flush(DataOutputStream.java:106)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendAckUpstreamUnprotected(BlockReceiver.java:1339)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver$PacketResponder.sendOOBResponse(BlockReceiver.java:1041)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:802)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:741)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:124)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:71)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:234)
at java.lang.Thread.run(Thread.java:695)
{noformat}
> OOB massage can't be sent to the client when DataNode shuts down for upgrade
>
>
> Key: HDFS-6569
> URL: https://issues.apache.org/jira/browse/HDFS-6569
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 3.0.0, 2.2.0
>Reporter: Brandon Li
>
> The socket is closed too early before the OOB message can be sent to client,
> which causes the write pipeline failure.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6569) OOB massage can't be sent to the client when DataNode shuts down for upgrade
Brandon Li created HDFS-6569: Summary: OOB massage can't be sent to the client when DataNode shuts down for upgrade Key: HDFS-6569 URL: https://issues.apache.org/jira/browse/HDFS-6569 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Brandon Li The socket is closed too early before the OOB message can be sent to client, which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6569) OOB massage can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6569: - Affects Version/s: 3.0.0 > OOB massage can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6569) OOB massage can't be sent to the client when DataNode shuts down for upgrade
[ https://issues.apache.org/jira/browse/HDFS-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-6569: - Affects Version/s: 2.2.0 > OOB massage can't be sent to the client when DataNode shuts down for upgrade > > > Key: HDFS-6569 > URL: https://issues.apache.org/jira/browse/HDFS-6569 > Project: Hadoop HDFS > Issue Type: Bug > Components: datanode >Affects Versions: 3.0.0, 2.2.0 >Reporter: Brandon Li > > The socket is closed too early before the OOB message can be sent to client, > which causes the write pipeline failure. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[
https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037617#comment-14037617
]
Hadoop QA commented on HDFS-3848:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651425/HDFS-3848-v2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7176//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7176//console
This message is automatically generated.
> A Bug in recoverLeaseInternal method of FSNameSystem class
> --
>
> Key: HDFS-3848
> URL: https://issues.apache.org/jira/browse/HDFS-3848
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: namenode
>Affects Versions: 0.23.1
>Reporter: Hooman Peiro Sajjad
> Labels: patch
> Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch
>
> Original Estimate: 1h
> Remaining Estimate: 1h
>
> This is a bug in logic of the method recoverLeaseInternal. In line 1322 it
> checks if the owner of the file is trying to recreate the file. The condition
> of the if statement is
> (leaseFile != null && leaseFile.equals(lease)) ||
> lease.getHolder().equals(holder)
> As it can be seen, there are two operands (conditions) connected with an "or"
> operator. The first operand is straight and will be true only if the holder
> of the file is the new holder. But the problem is the second operand which
> will be always true since the "lease" object is the one found by the "holder"
> by calling "Lease lease = leaseManager.getLease(holder);" in line 1315.
> To fix this I think the if statement only should contain the following the
> condition:
> (leaseFile != null && leaseFile.getHolder().equals(holder))
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Work started] (HDFS-6490) Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone
[ https://issues.apache.org/jira/browse/HDFS-6490?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-6490 started by Charles Lamb. > Fix the keyid format for generated keys in FSNamesystem.createEncryptionZone > - > > Key: HDFS-6490 > URL: https://issues.apache.org/jira/browse/HDFS-6490 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: namenode, security >Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) >Reporter: Charles Lamb >Assignee: Charles Lamb > > FSNamesystem.createEncryptionZone needs to create key ids with the format > hdfs://HOST:PORT/ -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037599#comment-14037599 ] Juan Yu commented on HDFS-6549: --- is this a typo? true> > Add support for accessing the NFS gateway from the AIX NFS client > - > > Key: HDFS-6549 > URL: https://issues.apache.org/jira/browse/HDFS-6549 > Project: Hadoop HDFS > Issue Type: Bug > Components: nfs >Affects Versions: 2.4.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-6549.patch, HDFS-6549.patch > > > We've identified two issues when trying to access the HDFS NFS Gateway from > an AIX NFS client: > # In the case of COMMITs, the AIX NFS client will always send 4096, or a > multiple of the page size, for the offset to be committed, even if fewer > bytes than this have ever, or will ever, be written to the file. This will > cause a write to a file from the AIX NFS client to hang on close unless the > size of that file is a multiple of 4096. > # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the > same cookie verifier for a given directory seemingly forever after that > directory is first accessed over NFS, instead of getting a new cookie > verifier for every set of incremental readdir calls. This means that if a > directory's mtime ever changes, the FS must be unmounted/remounted before > readdir calls on that dir from AIX will ever succeed again. > From my interpretation of RFC-1813, the NFS Gateway is in fact doing the > correct thing in both cases, but we can introduce simple changes on the NFS > Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission
[
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037600#comment-14037600
]
Hudson commented on HDFS-6375:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #5736 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/5736/])
Move HDFS-6375 down to 2.5 section in CHANGES.txt (wang:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603974)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
> Listing extended attributes with the search permission
> --
>
> Key: HDFS-6375
> URL: https://issues.apache.org/jira/browse/HDFS-6375
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: namenode
>Affects Versions: 3.0.0
>Reporter: Andrew Wang
>Assignee: Charles Lamb
> Fix For: 3.0.0, 2.5.0
>
> Attachments: HDFS-6375.1.patch, HDFS-6375.10.patch,
> HDFS-6375.11.patch, HDFS-6375.13.patch, HDFS-6375.2.patch, HDFS-6375.3.patch,
> HDFS-6375.4.patch, HDFS-6375.5.patch, HDFS-6375.6.patch, HDFS-6375.7.patch,
> HDFS-6375.8.patch, HDFS-6375.9.patch
>
>
> From the attr(5) manpage:
> {noformat}
>Users with search access to a file or directory may retrieve a list of
>attribute names defined for that file or directory.
> {noformat}
> This is like doing {{getfattr}} without the {{-d}} flag, which we currently
> don't support.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6492: -- Resolution: Fixed Fix Version/s: 2.5.0 Status: Resolved (was: Patch Available) Committed to trunk and branch-2. The branch-2 merge required some fixups because of HDFS-6315 being merged in while HDFS-2006 was only in trunk. > Support create-time xattrs and atomically setting multiple xattrs > - > > Key: HDFS-6492 > URL: https://issues.apache.org/jira/browse/HDFS-6492 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Fix For: 2.5.0 > > Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, > HDFS-6492.003.patch > > > Ongoing work in HDFS-6134 requires being able to set system namespace > extended attributes at create and mkdir time, as well as being able to > atomically set multiple xattrs at once. There's currently no need to expose > this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6557) Move the reference of fsimage to FSNamesystem
[
https://issues.apache.org/jira/browse/HDFS-6557?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6557:
-
Attachment: HDFS-6557.001.patch
> Move the reference of fsimage to FSNamesystem
> -
>
> Key: HDFS-6557
> URL: https://issues.apache.org/jira/browse/HDFS-6557
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Haohui Mai
>Assignee: Haohui Mai
> Attachments: HDFS-6557.000.patch, HDFS-6557.001.patch
>
>
> Per the suggestion from HDFS-6480. {{FSDirectory}} becomes a in-memory data
> structure so that the reference of fsimage should be moved to
> {{FSNamesystem}}.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14037576#comment-14037576 ] Hudson commented on HDFS-6492: -- FAILURE: Integrated in Hadoop-trunk-Commit #5735 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5735/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml > Support create-time xattrs and atomically setting multiple xattrs > - > > Key: HDFS-6492 > URL: https://issues.apache.org/jira/browse/HDFS-6492 > Project: Hadoop HDFS > Issue Type: Improvement > Components: namenode >Affects Versions: 2.4.0 >Reporter: Andrew Wang >Assignee: Andrew Wang > Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, > HDFS-6492.003.patch > > > Ongoing work in HDFS-6134 requires being able to set system namespace > extended attributes at create and mkdir time, as well as being able to > atomically set multiple xattrs at once. There's currently no need to expose > this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
