[jira] [Assigned] (HDFS-3269) End-to-end test for making a non-HA HDFS cluster HA-enabled
[ https://issues.apache.org/jira/browse/HDFS-3269?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Mingjie Lai reassigned HDFS-3269: - Assignee: Mingjie Lai End-to-end test for making a non-HA HDFS cluster HA-enabled --- Key: HDFS-3269 URL: https://issues.apache.org/jira/browse/HDFS-3269 Project: Hadoop HDFS Issue Type: Test Components: ha, name-node Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Mingjie Lai Priority: Minor Per Eli on HDFS-3259, it would be great if we had a test that did the following: # Starts w/ non HA NN1 # Shutdown, enable HA on NN1, add SBN NN2 # Run initializeSharedEdits # Start and transition to active NN1 # Run bootstrapStandby # Confirm NN1 and NN2 are up and HA -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3282) Expose getFileLength API.
Expose getFileLength API. - Key: HDFS-3282 URL: https://issues.apache.org/jira/browse/HDFS-3282 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G This JIRA is to expose the getFileLength API through a new public DistributedFileSystemInfo class. I would appreciate if someone suggest good name for this public class. Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2704) NameNodeResouceChecker#checkAvailableResources should check for inodes
[ https://issues.apache.org/jira/browse/HDFS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254700#comment-13254700 ] Robert Joseph Evans commented on HDFS-2704: --- Good point Aaron :) NameNodeResouceChecker#checkAvailableResources should check for inodes -- Key: HDFS-2704 URL: https://issues.apache.org/jira/browse/HDFS-2704 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.24.0 Reporter: Eli Collins Assignee: SreeHari NameNodeResouceChecker#checkAvailableResources currently just checks for free space. However inodes are also a file system resource that needs to be available (you can run out of inodes but have free space). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3283) create http server for JournalNode
create http server for JournalNode -- Key: HDFS-3283 URL: https://issues.apache.org/jira/browse/HDFS-3283 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Brandon Li Assignee: Brandon Li This http server can be used to view journal node status and transfer finalized edit log segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3283) create http server for JournalNode
[ https://issues.apache.org/jira/browse/HDFS-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3283: - Status: Patch Available (was: Open) create http server for JournalNode -- Key: HDFS-3283 URL: https://issues.apache.org/jira/browse/HDFS-3283 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-3283.HDFS-3092.patch This http server can be used to view journal node status and transfer finalized edit log segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3283) create http server for JournalNode
[ https://issues.apache.org/jira/browse/HDFS-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3283: - Attachment: HDFS-3283.HDFS-3092.patch create http server for JournalNode -- Key: HDFS-3283 URL: https://issues.apache.org/jira/browse/HDFS-3283 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Brandon Li Assignee: Brandon Li Attachments: HDFS-3283.HDFS-3092.patch This http server can be used to view journal node status and transfer finalized edit log segments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3284) bootstrapStandby fails in secure cluster
bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3284: -- Attachment: hdfs-3284.txt Simple patch to fix this. I tested manually on a secure cluster and verified the problem is fixed. No unit tests because I can't get the new secure test infrastructure to actually work as advertised. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254889#comment-13254889 ] dhruba borthakur commented on HDFS-3092: I still somehow like the BK approach, especially because it can store transaction logs from multiple namenodes, this is something that is very needed for HDFS-federation. If we can get BK to scale and be performant, it is a possibility that HBase can store transaction logs in there too, especially of the latencies are much smaller (and disk-failure recoveries faster) than a typical HDFS write pipeline. Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254906#comment-13254906 ] Aaron T. Myers commented on HDFS-3284: -- +1, the patch looks good to me. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3284: -- Status: Patch Available (was: Open) bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3268) Hdfs mishandles token service incompatible with HA
[
https://issues.apache.org/jira/browse/HDFS-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254916#comment-13254916
]
Todd Lipcon commented on HDFS-3268:
---
+1, will commit this momentarily. Thanks, Daryn.
Hdfs mishandles token service incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Attachments: HDFS-3268-1.patch, HDFS-3268.patch
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this
improper behavior is overwriting the HA logical service which indirectly
renders {{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254930#comment-13254930 ] Hadoop QA commented on HDFS-3284: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522829/hdfs-3284.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The patch appears to cause tar ant target to fail. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2280//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2280//console This message is automatically generated. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3268) Hdfs mishandles token service incompatible with HA
[
https://issues.apache.org/jira/browse/HDFS-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3268:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-2 and trunk. Thanks Daryn.
Hdfs mishandles token service incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Fix For: 2.0.0
Attachments: HDFS-3268-1.patch, HDFS-3268.patch
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this
improper behavior is overwriting the HA logical service which indirectly
renders {{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254934#comment-13254934 ] Todd Lipcon commented on HDFS-3284: --- ah, the addSecurityConfiguration function is only on the auto-HA branch. Let me pull that into this patch as well. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3268) Hdfs mishandles token service incompatible with HA
[
https://issues.apache.org/jira/browse/HDFS-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254939#comment-13254939
]
Hudson commented on HDFS-3268:
--
Integrated in Hadoop-Common-trunk-Commit #2078 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2078/])
HDFS-3268. FileContext API mishandles token service and incompatible with
HA. Contributed by Daryn Sharp. (Revision 1326747)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326747
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
Hdfs mishandles token service incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Fix For: 2.0.0
Attachments: HDFS-3268-1.patch, HDFS-3268.patch
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this
improper behavior is overwriting the HA logical service which indirectly
renders {{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254941#comment-13254941 ] Tsz Wo (Nicholas), SZE commented on HDFS-3282: -- Hi Uma, We could move DFSClient.DFSDataInputStream to a new class, say org.apache.hadoop.hdfs.client.HdfsDataInputStream, which is a public API. Then, change the return type of DistributedFileSystem.open(..) to HdfsDataInputStream. What do you think? Expose getFileLength API. - Key: HDFS-3282 URL: https://issues.apache.org/jira/browse/HDFS-3282 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G This JIRA is to expose the getFileLength API through a new public DistributedFileSystemInfo class. I would appreciate if someone suggest good name for this public class. Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3268) Hdfs mishandles token service incompatible with HA
[
https://issues.apache.org/jira/browse/HDFS-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254944#comment-13254944
]
Hudson commented on HDFS-3268:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2151 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2151/])
HDFS-3268. FileContext API mishandles token service and incompatible with
HA. Contributed by Daryn Sharp. (Revision 1326747)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326747
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
Hdfs mishandles token service incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Fix For: 2.0.0
Attachments: HDFS-3268-1.patch, HDFS-3268.patch
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this
improper behavior is overwriting the HA logical service which indirectly
renders {{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3268) Hdfs mishandles token service incompatible with HA
[
https://issues.apache.org/jira/browse/HDFS-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254946#comment-13254946
]
Hudson commented on HDFS-3268:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2092 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2092/])
HDFS-3268. FileContext API mishandles token service and incompatible with
HA. Contributed by Daryn Sharp. (Revision 1326747)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326747
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/Hdfs.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSClient.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
Hdfs mishandles token service incompatible with HA
Key: HDFS-3268
URL: https://issues.apache.org/jira/browse/HDFS-3268
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, hdfs client
Affects Versions: 0.24.0, 2.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Priority: Critical
Fix For: 2.0.0
Attachments: HDFS-3268-1.patch, HDFS-3268.patch
The {{Hdfs AbstractFileSystem}} is overwriting the token service set by the
{{DFSClient}}. The service is not necessarily the correct one since
{{DFSClient}} is responsible for the service. Most importantly, this
improper behavior is overwriting the HA logical service which indirectly
renders {{FileContext}} incompatible with HA.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3279) One of the FSEditLog constructors should be moved to TestEditLog
[ https://issues.apache.org/jira/browse/HDFS-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3279: - Summary: One of the FSEditLog constructors should be moved to TestEditLog (was: One of the FSEditLog constructor should be moved to TestEditLog) Hadoop Flags: Reviewed +1 patch looks good. One of the FSEditLog constructors should be moved to TestEditLog Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 3.0.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor Attachments: HDFS-3279.patch The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3279) One of the FSEditLog constructors should be moved to TestEditLog
[ https://issues.apache.org/jira/browse/HDFS-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3279: - Resolution: Fixed Fix Version/s: 2.0.0 Status: Resolved (was: Patch Available) I have committed this. Thanks, Arpit! One of the FSEditLog constructors should be moved to TestEditLog Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 3.0.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor Fix For: 2.0.0 Attachments: HDFS-3279.patch The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2652) Port token service changes from 205
[
https://issues.apache.org/jira/browse/HDFS-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daryn Sharp updated HDFS-2652:
--
Attachment: HDFS-2652.patch
Implements remaining changes to make hdfs compliant with host-based tokens.
Notable changes:
* {{HAUtil.cloneDelegationTokenForLogicalUri}}
** Takes a collection of all NN addresses instead of one. The logical token is
looked up once and duped to all NNs in once pass.
** Does not call the static token selector method that rewrites the service
port (see below).
** Changed to use {{SecurityUtil.setTokenService}}.
* Hftp and WebHDFS token selectors are:
** Cleaned up to look for their own token and fall thru to a hdfs lookup with a
rewritten port.
** URIs are used to ensure the original host survives the rpc port rewrite.
* {{selectHdfsDelegationToken}} rewrites service's port to lookup rpc tokens
for a remote cluster.
** Replaced {{InetSocketAddress}} arg with a URI to ensure the original host in
the URI is preserved when the address is reconstructed.
** The method is only applicable for hftp and webhdfs, so HA no longer uses the
method.
** Signature made more compliant with other select method.
* Bunch of tests
Port token service changes from 205
---
Key: HDFS-2652
URL: https://issues.apache.org/jira/browse/HDFS-2652
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 0.24.0, 0.23.1
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-2652.patch
Need to merge the 205 token bug fixes and the feature to enable
hostname-based tokens. See jiras linked to HADOOP-7808 for more details.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2652) Port token service changes from 205
[ https://issues.apache.org/jira/browse/HDFS-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Daryn Sharp updated HDFS-2652: -- Target Version/s: 0.23.3 (was: 0.23.1, 0.24.0) Status: Patch Available (was: Open) Current patch is for trunk, currently testing against other branches. Port token service changes from 205 --- Key: HDFS-2652 URL: https://issues.apache.org/jira/browse/HDFS-2652 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.23.1, 0.24.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-2652.patch Need to merge the 205 token bug fixes and the feature to enable hostname-based tokens. See jiras linked to HADOOP-7808 for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3273: - Description: Some code in BackupImage keep referencing FSEditLog. It is better to move those code to FSEditLog. Aaron, when creating a sub-task in the main task, JIRA does not let us adding description. Refactor BackupImage and FSEditLog -- Key: HDFS-3273 URL: https://issues.apache.org/jira/browse/HDFS-3273 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 3.0.0 Attachments: h3273_20120413.patch Some code in BackupImage keep referencing FSEditLog. It is better to move those code to FSEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254967#comment-13254967 ] Aaron T. Myers commented on HDFS-3273: -- bq. when creating a sub-task in the main task, JIRA does not let us adding description. That only happens if you click the + on the parent JIRA. If you go to the parent JIRA - More Actions - Create Sub-Task, you'll get a big form with all the normal JIRA fields. Refactor BackupImage and FSEditLog -- Key: HDFS-3273 URL: https://issues.apache.org/jira/browse/HDFS-3273 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 3.0.0 Attachments: h3273_20120413.patch Some code in BackupImage keep referencing FSEditLog. It is better to move those code to FSEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3282) Expose getFileLength API.
[ https://issues.apache.org/jira/browse/HDFS-3282?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254963#comment-13254963 ] Todd Lipcon commented on HDFS-3282: --- Nicholas: despite us advertising DFSDataInputStream as a private API, I imagine this change would break people. Could we instead just add a new interface which would be implemented by the existing class? Expose getFileLength API. - Key: HDFS-3282 URL: https://issues.apache.org/jira/browse/HDFS-3282 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs client Affects Versions: 3.0.0 Reporter: Uma Maheswara Rao G Assignee: Uma Maheswara Rao G This JIRA is to expose the getFileLength API through a new public DistributedFileSystemInfo class. I would appreciate if someone suggest good name for this public class. Nicholas, did you plan any special design for this public client class? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3279) One of the FSEditLog constructors should be moved to TestEditLog
[ https://issues.apache.org/jira/browse/HDFS-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254972#comment-13254972 ] Hudson commented on HDFS-3279: -- Integrated in Hadoop-Hdfs-trunk-Commit #2152 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2152/]) HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to TestEditLog. Contributed by Arpit Gupta (Revision 1326762) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326762 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java One of the FSEditLog constructors should be moved to TestEditLog Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 3.0.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor Fix For: 2.0.0 Attachments: HDFS-3279.patch The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Status: Patch Available (was: Open) HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254980#comment-13254980 ] Tsz Wo (Nicholas), SZE commented on HDFS-3273: -- Sure, you could also edit the JIRA. Refactor BackupImage and FSEditLog -- Key: HDFS-3273 URL: https://issues.apache.org/jira/browse/HDFS-3273 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 3.0.0 Attachments: h3273_20120413.patch Some code in BackupImage keep referencing FSEditLog. It is better to move those code to FSEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Attachment: HDFS-3263.patch Now HttpFS, by default, reads the HDFS configuration from the httpfs conf/ dir, which in the TARBALL is the same as for the rest of the Hadoop services. HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3279) One of the FSEditLog constructors should be moved to TestEditLog
[ https://issues.apache.org/jira/browse/HDFS-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13254985#comment-13254985 ] Hudson commented on HDFS-3279: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2093 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2093/]) HDFS-3279. Move the FSEditLog constructor with @VisibleForTesting to TestEditLog. Contributed by Arpit Gupta (Revision 1326762) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326762 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestEditLog.java One of the FSEditLog constructors should be moved to TestEditLog Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 3.0.0 Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor Fix For: 2.0.0 Attachments: HDFS-3279.patch The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3284: -- Attachment: hdfs-3284.txt Patch which should work on trunk (re-tested with a secure trunk HA cluster) bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from recovery should be removed from DN.
[ https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255026#comment-13255026 ] Todd Lipcon commented on HDFS-3161: --- Hi Uma/Vinay. I ran into an issue like this without use of append(): - Client writing blk_N_GS1 to DN1, DN9, DN10 - Pipeline failed. commitBlockSynchronization succeeded with DN9 and DN10, sets gs to blk_N_GS2 - Client closes the pipeline - NN issues replication request of blk_N_GS2 from DN9 to DN1 - DN1 already has blk_N_GS1 in its ongoingCreates map I'm not sure if this can cause any serious issue with the block (it didn't in my case), but I agree that, if a replication request happens for a block with a higher genstamp, it should interrupt the old block's ongoingCreate. If the replication request is a lower genstamp, it should be ignored. 20 Append: Excluded DN replica from recovery should be removed from DN. --- Key: HDFS-3161 URL: https://issues.apache.org/jira/browse/HDFS-3161 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.0 Reporter: suja s Priority: Critical Fix For: 1.0.3 1) DN1-DN2-DN3 are in pipeline. 2) Client killed abruptly 3) one DN has restarted , say DN3 4) In DN3 info.wasRecoveredOnStartup() will be true 5) NN recovery triggered, DN3 skipped from recovery due to above check. 6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older generation stamp say 1 and also DN3 still has this block entry in ongoingCreates 7) as part of recovery file has closed and got only two live replicas ( from DN1 and DN2) 8) So, NN issued the command for replication. Now DN3 also has the replica with newer generation stamp. 9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with referring to blocksBeingWritten directory. When we call append/ leaseRecovery, it may again skip this node for that recovery as blockId entry still presents in ongoingCreates with startup recovery true. It may keep continue this dance for evry recovery. And this stale replica will not be cleaned untill we restart the cluster. Actual replica will be trasferred to this node only through replication process. Also unnecessarily that replicated blocks will get invalidated after next recoveries -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2652) Port token service changes from 205
[ https://issues.apache.org/jira/browse/HDFS-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255039#comment-13255039 ] Hadoop QA commented on HDFS-2652: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522842/HDFS-2652.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 5 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2281//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2281//console This message is automatically generated. Port token service changes from 205 --- Key: HDFS-2652 URL: https://issues.apache.org/jira/browse/HDFS-2652 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 0.23.1 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-2652.patch Need to merge the 205 token bug fixes and the feature to enable hostname-based tokens. See jiras linked to HADOOP-7808 for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[
https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255047#comment-13255047
]
Daryn Sharp commented on HDFS-3263:
---
I've only skimmed the patch. I'm curious if there's a reason why it seems to
be hard coding keys instead of using the keys from
{{CommonConfigurationKeysPublic}}, etc? Ex. fs.default.name, which appears
to be deprecated, instead of {{FS_DEFAULT_NAME_KEY}}. Also, why does it need
to hardcode the resource files for {{Configuration}}?
HttpFS should read HDFS config from Hadoop site.xml files
-
Key: HDFS-3263
URL: https://issues.apache.org/jira/browse/HDFS-3263
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 2.0.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Fix For: 2.0.0
Attachments: HDFS-3263.patch
Currently HttpFS reads HDFS client configuration from the httfs-site.xml from
any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY'
This is a bit inconvenient.
Instead we should support a single property 'httpfs.hadoop.configuration.dir'
that can be pointed to HADOOP conf/ dir and the core-site.xml and
hdfs-site.xml files would be read from there.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255058#comment-13255058 ] Alejandro Abdelnur commented on HDFS-3263: -- @Daryn, the patch does not introduce the use of 'fs.default.name', it was already used. Only testcases adding new references. We coulld get rid of all those in another JIRA. Regarding the resource files (core-site.xml hdfs-site.xml) being hardcoded, httpfs is a webapp, it does not have hadoop conf/ dir in the classpath, that is why. thxs HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2652) Port token service changes from 205
[ https://issues.apache.org/jira/browse/HDFS-2652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255080#comment-13255080 ] Daryn Sharp commented on HDFS-2652: --- branch-2 seems fine. will need to update for 23 to remove HA related patches Port token service changes from 205 --- Key: HDFS-2652 URL: https://issues.apache.org/jira/browse/HDFS-2652 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 0.24.0, 0.23.1 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-2652.patch Need to merge the 205 token bug fixes and the feature to enable hostname-based tokens. See jiras linked to HADOOP-7808 for more details. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255098#comment-13255098 ] Hadoop QA commented on HDFS-3284: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522849/hdfs-3284.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestListCorruptFileBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2282//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2282//console This message is automatically generated. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255101#comment-13255101 ] Todd Lipcon commented on HDFS-3284: --- The test failure is unrelated (this patch doesn't touch that area of the code) bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255103#comment-13255103 ] Aaron T. Myers commented on HDFS-3284: -- +1, the updated patch looks good to me. Thanks a lot for taking care of this, Todd. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3284: -- Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to trunk and branch-2. bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 2.0.0 Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255112#comment-13255112 ] Hadoop QA commented on HDFS-3263: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522848/HDFS-3263.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.fs.http.server.TestHttpFSServer +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2283//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2283//console This message is automatically generated. HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255117#comment-13255117 ] Hudson commented on HDFS-3284: -- Integrated in Hadoop-Common-trunk-Commit #2082 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2082/]) HDFS-3284. bootstrapStandby fails in secure cluster. Contributed by Todd Lipcon. (Revision 1326813) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 2.0.0 Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255118#comment-13255118 ] Hudson commented on HDFS-3284: -- Integrated in Hadoop-Hdfs-trunk-Commit #2155 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2155/]) HDFS-3284. bootstrapStandby fails in secure cluster. Contributed by Todd Lipcon. (Revision 1326813) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 2.0.0 Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3049:
---
Attachment: HDFS-3049.002.patch
* fix tests
During the normal loading NN startup process, fall back on a different
EditLog if we see one that is corrupt
Key: HDFS-3049
URL: https://issues.apache.org/jira/browse/HDFS-3049
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Affects Versions: 0.23.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-3049.001.patch, HDFS-3049.002.patch
During the NameNode startup process, we load an image, and then apply edit
logs to it until we believe that we have all the latest changes.
Unfortunately, if there is an I/O error while reading any of these files, in
most cases, we simply abort the startup process. We should try harder to
locate a readable edit log and/or image file.
*There are three main use cases for this feature:*
1. If the operating system does not honor fsync (usually due to a
misconfiguration), a file may end up in an inconsistent state.
2. In certain older releases where we did not use fallocate() or similar to
pre-reserve blocks, a disk full condition may cause a truncated log in one
edit directory.
3. There may be a bug in HDFS which results in some of the data directories
receiving corrupt data, but not all. This is the least likely use case.
*Proposed changes to normal NN startup*
* We should try a different FSImage if we can't load the first one we try.
* We should examine other FSEditLogs if we can't load the first one(s) we try.
* We should fail if we can't find EditLogs that would bring us up to what we
believe is the latest transaction ID.
Proposed changes to recovery mode NN startup:
we should list out all the available storage directories and allow the
operator to select which one he wants to use.
Something like this:
{code}
Multiple storage directories found.
1. /foo/bar
edits__curent__XYZ size:213421345 md5:2345345
image size:213421345 md5:2345345
2. /foo/baz
edits__curent__XYZ size:213421345 md5:2345345345
image size:213421345 md5:2345345
Which one would you like to use? (1/2)
{code}
As usual in recovery mode, we want to be flexible about error handling. In
this case, this means that we should NOT fail if we can't find EditLogs that
would bring us up to what we believe is the latest transaction ID.
*Not addressed by this feature*
This feature will not address the case where an attempt to access the
NameNode name directory or directories hangs because of an I/O error. This
may happen, for example, when trying to load an image from a hard-mounted NFS
directory, when the NFS server has gone away. Just as now, the operator will
have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255159#comment-13255159
]
Hadoop QA commented on HDFS-3049:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522871/HDFS-3049.002.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 4 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 javac. The patch appears to cause tar ant target to fail.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to cause Findbugs (version 1.3.9) to fail.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2284//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2284//console
This message is automatically generated.
During the normal loading NN startup process, fall back on a different
EditLog if we see one that is corrupt
Key: HDFS-3049
URL: https://issues.apache.org/jira/browse/HDFS-3049
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Affects Versions: 0.23.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Priority: Minor
Attachments: HDFS-3049.001.patch, HDFS-3049.002.patch
During the NameNode startup process, we load an image, and then apply edit
logs to it until we believe that we have all the latest changes.
Unfortunately, if there is an I/O error while reading any of these files, in
most cases, we simply abort the startup process. We should try harder to
locate a readable edit log and/or image file.
*There are three main use cases for this feature:*
1. If the operating system does not honor fsync (usually due to a
misconfiguration), a file may end up in an inconsistent state.
2. In certain older releases where we did not use fallocate() or similar to
pre-reserve blocks, a disk full condition may cause a truncated log in one
edit directory.
3. There may be a bug in HDFS which results in some of the data directories
receiving corrupt data, but not all. This is the least likely use case.
*Proposed changes to normal NN startup*
* We should try a different FSImage if we can't load the first one we try.
* We should examine other FSEditLogs if we can't load the first one(s) we try.
* We should fail if we can't find EditLogs that would bring us up to what we
believe is the latest transaction ID.
Proposed changes to recovery mode NN startup:
we should list out all the available storage directories and allow the
operator to select which one he wants to use.
Something like this:
{code}
Multiple storage directories found.
1. /foo/bar
edits__curent__XYZ size:213421345 md5:2345345
image size:213421345 md5:2345345
2. /foo/baz
edits__curent__XYZ size:213421345 md5:2345345345
image size:213421345 md5:2345345
Which one would you like to use? (1/2)
{code}
As usual in recovery mode, we want to be flexible about error handling. In
this case, this means that we should NOT fail if we can't find EditLogs that
would bring us up to what we believe is the latest transaction ID.
*Not addressed by this feature*
This feature will not address the case where an attempt to access the
NameNode name directory or directories hangs because of an I/O error. This
may happen, for example, when trying to load an image from a hard-mounted NFS
directory, when the NFS server has gone away. Just as now, the operator will
have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3284) bootstrapStandby fails in secure cluster
[ https://issues.apache.org/jira/browse/HDFS-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255171#comment-13255171 ] Hudson commented on HDFS-3284: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2096 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2096/]) HDFS-3284. bootstrapStandby fails in secure cluster. Contributed by Todd Lipcon. (Revision 1326813) Result = ABORTED todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326813 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/BootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DFSHAAdmin.java bootstrapStandby fails in secure cluster Key: HDFS-3284 URL: https://issues.apache.org/jira/browse/HDFS-3284 Project: Hadoop HDFS Issue Type: Bug Components: ha, security Affects Versions: 2.0.0 Reporter: Todd Lipcon Assignee: Todd Lipcon Priority: Minor Fix For: 2.0.0 Attachments: hdfs-3284.txt, hdfs-3284.txt HDFS-3247 improved bootstrapStandby to check if the other NN is in active state before trying to bootstrap. But, it forgot to set up the kerberos principals in the config before doing so. So, bootstrapStandby now fails with Failed to specify server's Kerberos principal name in a secure cluster. (Credit to Stephen Chu for finding this) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2246) Shortcut a local client reads to a Datanodes files directly
[ https://issues.apache.org/jira/browse/HDFS-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Benoy Antony updated HDFS-2246: --- Attachment: HDFS-2246-22.patch The attached patch for 22 is ported from patch for 205. Shortcut a local client reads to a Datanodes files directly --- Key: HDFS-2246 URL: https://issues.apache.org/jira/browse/HDFS-2246 Project: Hadoop HDFS Issue Type: Improvement Reporter: Sanjay Radia Assignee: Jitendra Nath Pandey Fix For: 0.23.1, 1.0.0 Attachments: 0001-HDFS-347.-Local-reads.patch, HDFS-2246-22.patch, HDFS-2246-branch-0.20-security-205.1.patch, HDFS-2246-branch-0.20-security-205.2.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security-205.patch, HDFS-2246-branch-0.20-security.3.patch, HDFS-2246-branch-0.20-security.no-softref.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-branch-0.20-security.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246-trunk.patch, HDFS-2246.20s.1.patch, HDFS-2246.20s.2.txt, HDFS-2246.20s.3.txt, HDFS-2246.20s.4.txt, HDFS-2246.20s.patch, TestShortCircuitLocalRead.java, localReadShortcut20-security.2patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2631) Rewrite fuse-dfs to use the webhdfs protocol
[ https://issues.apache.org/jira/browse/HDFS-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jaimin D Jetly reassigned HDFS-2631: Assignee: Jaimin D Jetly Rewrite fuse-dfs to use the webhdfs protocol Key: HDFS-2631 URL: https://issues.apache.org/jira/browse/HDFS-2631 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs Reporter: Eli Collins Assignee: Jaimin D Jetly We should port the implementation of fuse-dfs to use the webhdfs protocol. This has a number of benefits: * Compatibility - allows a single fuse client to work across server versions * Works with both WebHDFS and Hoop since they are protocol compatible * Removes the overhead related to libhdfs (forking a jvm) * Makes it easier to support features like security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-3165: - Assignee: amith HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0, 3.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Attachment: HDFS-3263.patch fixing instrumentation testcase to use testcases injected users/groups instead OS ones for root as they seem to differ in different Unix HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch, HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3165: -- Resolution: Fixed Status: Resolved (was: Patch Available) I've committed this and merged to branch-2. Thanks Amith! HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3165: -- Target Version/s: 2.0.0 (was: 3.0.0, 2.0.0) Affects Version/s: (was: 3.0.0) Hadoop Flags: Reviewed +1, looks good. I built a tarball and verified that the balancer now runs via the start script. HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255232#comment-13255232 ] Hudson commented on HDFS-3165: -- Integrated in Hadoop-Hdfs-trunk-Commit #2156 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2156/]) HDFS-3165. HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh. Contributed by Amith D K (Revision 1326848) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326848 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255235#comment-13255235 ] Hudson commented on HDFS-3165: -- Integrated in Hadoop-Common-trunk-Commit #2083 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2083/]) HDFS-3165. HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh. Contributed by Amith D K (Revision 1326848) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326848 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255243#comment-13255243 ] Hudson commented on HDFS-3165: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2097 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2097/]) HDFS-3165. HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh. Contributed by Amith D K (Revision 1326848) Result = ABORTED eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1326848 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/start-balancer.sh * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/bin/stop-balancer.sh HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: scripts Affects Versions: 2.0.0 Environment: HDFS Balancer Reporter: amith Assignee: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch, HDFS-3165.patch, HDFS-3165_1.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2631) Rewrite fuse-dfs to use the webhdfs protocol
[ https://issues.apache.org/jira/browse/HDFS-2631?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255255#comment-13255255 ] Todd Lipcon commented on HDFS-2631: --- I'm a little confused: why is this a good idea? Seems like it's likely to end up much slower than the current implementation. I'd prefer it as another option, rather than a rewrite. Rewrite fuse-dfs to use the webhdfs protocol Key: HDFS-2631 URL: https://issues.apache.org/jira/browse/HDFS-2631 Project: Hadoop HDFS Issue Type: Improvement Components: contrib/fuse-dfs Reporter: Eli Collins Assignee: Jaimin D Jetly We should port the implementation of fuse-dfs to use the webhdfs protocol. This has a number of benefits: * Compatibility - allows a single fuse client to work across server versions * Works with both WebHDFS and Hoop since they are protocol compatible * Removes the overhead related to libhdfs (forking a jvm) * Makes it easier to support features like security -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255257#comment-13255257 ] Hadoop QA commented on HDFS-3263: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522893/HDFS-3263.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. -1 javac. The applied patch generated 21 javac compiler warnings (more than the trunk's current 20 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2285//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2285//console This message is automatically generated. HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch, HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-3263: - Attachment: HDFS-3263.patch removing spurious import that generated javac warnings HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3161) 20 Append: Excluded DN replica from recovery should be removed from DN.
[
https://issues.apache.org/jira/browse/HDFS-3161?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255269#comment-13255269
]
Uma Maheswara Rao G commented on HDFS-3161:
---
Todd, This situation can occur, but we have seen the problem only in append, as
it will skip the recoveries if entry presents in ongoingCreates.
In this case:
bq. DN1 already has blk_N_GS1 in its ongoingCreates map
Block transfer will happen successfully as part of replication. Also reader
should be able to read it with newer genstamp. If there is no append for this
file, then there won't be any recoveries. So, we need not worry about skipping
the DN from recovery if it presents in ongoing creates.
Only thing is, that block and ongoing creates entry will not be cleared until
we restart the cluster.
Anyway let me confirm whether readers are able to read properly or not. And
will write a test with your scenario.
Do you think any other problems?
{quote}
but I agree that, if a replication request happens for a block with a higher
genstamp, it should interrupt the old block's ongoingCreate. If the replication
request is a lower genstamp, it should be ignored.
{quote}
If we really want to address this case, then this would be the fix I feel.
20 Append: Excluded DN replica from recovery should be removed from DN.
---
Key: HDFS-3161
URL: https://issues.apache.org/jira/browse/HDFS-3161
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 1.0.0
Reporter: suja s
Priority: Critical
Fix For: 1.0.3
1) DN1-DN2-DN3 are in pipeline.
2) Client killed abruptly
3) one DN has restarted , say DN3
4) In DN3 info.wasRecoveredOnStartup() will be true
5) NN recovery triggered, DN3 skipped from recovery due to above check.
6) Now DN1, DN2 has blocks with generataion stamp 2 and DN3 has older
generation stamp say 1 and also DN3 still has this block entry in
ongoingCreates
7) as part of recovery file has closed and got only two live replicas ( from
DN1 and DN2)
8) So, NN issued the command for replication. Now DN3 also has the replica
with newer generation stamp.
9) Now DN3 contains 2 replicas on disk. and one entry in ongoing creates with
referring to blocksBeingWritten directory.
When we call append/ leaseRecovery, it may again skip this node for that
recovery as blockId entry still presents in ongoingCreates with startup
recovery true.
It may keep continue this dance for evry recovery.
And this stale replica will not be cleaned untill we restart the cluster.
Actual replica will be trasferred to this node only through replication
process.
Also unnecessarily that replicated blocks will get invalidated after next
recoveries
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3196) Implement JournalListener for writing journal to local disk
[ https://issues.apache.org/jira/browse/HDFS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3196: - Attachment: h3196_20120416.patch h3196_20120416.patch: adds a Journal class so that reader and writer can use it. Implement JournalListener for writing journal to local disk --- Key: HDFS-3196 URL: https://issues.apache.org/jira/browse/HDFS-3196 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3196_20120412.patch, h3196_20120416.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3263) HttpFS should read HDFS config from Hadoop site.xml files
[ https://issues.apache.org/jira/browse/HDFS-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255293#comment-13255293 ] Hadoop QA commented on HDFS-3263: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522902/HDFS-3263.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 4 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2286//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2286//console This message is automatically generated. HttpFS should read HDFS config from Hadoop site.xml files - Key: HDFS-3263 URL: https://issues.apache.org/jira/browse/HDFS-3263 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Fix For: 2.0.0 Attachments: HDFS-3263.patch, HDFS-3263.patch, HDFS-3263.patch Currently HttpFS reads HDFS client configuration from the httfs-site.xml from any property of the form 'httpfs.hadoop.conf:HADOOP_PROPERTY' This is a bit inconvenient. Instead we should support a single property 'httpfs.hadoop.configuration.dir' that can be pointed to HADOOP conf/ dir and the core-site.xml and hdfs-site.xml files would be read from there. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13255340#comment-13255340 ] Allen Wittenauer commented on HDFS-3271: Or test for POSIX compliance at configure time... (FWIW, I'm still mostly convinced that this ugly hack was a waste of time for the majority of folks.) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc -- Key: HDFS-3271 URL: https://issues.apache.org/jira/browse/HDFS-3271 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
