[jira] [Commented] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246058#comment-13246058 ] Suresh Srinivas commented on HDFS-3178: --- Nicholas, if you are going to check for all states in StateMachine, you might as well move that code into JournalService. Rest of the code looks good. Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246059#comment-13246059 ] Suresh Srinivas commented on HDFS-3192: --- Hari, agree with Aaron that this should not be a subtask of HDFS-3092. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3193) HDFS journal logging improvement
HDFS journal logging improvement Key: HDFS-3193 URL: https://issues.apache.org/jira/browse/HDFS-3193 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Suresh Srinivas I want to use this jira to discuss doing HDFS editlog logging before namespace update. Couple of bugs (though have been solved with work arounds) HDFS-2815, HDFS-3191 are related to this. Also currently an update to namespace becomes visible to other readers, even though it is not persisted in editlog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3183) Add JournalManager implementation to JournalDaemons for storing edits
[ https://issues.apache.org/jira/browse/HDFS-3183?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-3183: -- Description: The JournalManager is used in HA configuration and uses the following journal daemons on: - Local namenode - Other namenode - A configured JournalDaemon target from the configuration was: The JournalManager is used in HA configuration and uses the following journal targets: - local namenode - Other namenode - A configured JournalDaemon target from configuration Summary: Add JournalManager implementation to JournalDaemons for storing edits (was: Add JournalManager implementation to use local namenode, remote namenode and a configured JournalDaemon for storing editlogs) Add JournalManager implementation to JournalDaemons for storing edits - Key: HDFS-3183 URL: https://issues.apache.org/jira/browse/HDFS-3183 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Suresh Srinivas The JournalManager is used in HA configuration and uses the following journal daemons on: - Local namenode - Other namenode - A configured JournalDaemon target from the configuration -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[
https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3084:
--
Attachment: hdfs-3084.txt
This patch, in conjunction with HADOOP-8007, provides the necessary facilities:
- the configuration used to instantiate the NodeFencer is now the configuration
of the _target_ node, rather than the _fencing_ node. This means that all of
the configs like RPC address have already had the nameservice and namenodeid
chopped off
- in addition to the variables described in HADOOP-8007, we add
$target_namenodeid and $target_nameserviceid to the fencing environment
- added getters for namenode id and nameservice id to NNHAServiceTarget. If a
custom fencing implementation is HDFS specific, it can downcast to the
implementation class to access these getters. Alternatively, it may call
{{getFencingParameters()}} and then use the resulting string map to get at this
info.
The patch depends on HADOOP-8007. I'll submit it to QA once that other patch is
committed.
FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well
as host:port
-
Key: HDFS-3084
URL: https://issues.apache.org/jira/browse/HDFS-3084
Project: Hadoop HDFS
Issue Type: Improvement
Components: ha
Affects Versions: 0.24.0, 0.23.3
Reporter: Philip Zeyliger
Assignee: Todd Lipcon
Attachments: hdfs-3084.txt
The FenceMethod interface passes along the host:port of the NN that needs to
be fenced. That's great for the common case. However, it's likely necessary
to have extra configuration parameters for fencing, and these are typically
keyed off the nameserviceId.namenodeId (if, for nothing else, consistency
with all the other parameters that are keyed off of namespaceId.namenodeId).
Obviously this can be backed out from the host:port, but it's inconvenient,
and requires iterating through all the configs.
The shell interface exhibits the same issue: host:port is great for most
fencers, but if you need extra configs (like the host:port of the power
supply unit), those are harder to pipe through without the namenodeId.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246087#comment-13246087 ] Aaron T. Myers commented on HDFS-3084: -- +1, the patch looks good to me. Good stuff, Todd. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246086#comment-13246086 ] Eli Collins commented on HDFS-3084: --- +1 looks good FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3102: - Attachment: HDFS-3102.patch Here's a patch rebased on trunk. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3084: -- Target Version/s: 0.24.0, 0.23.3 (was: 0.23.3, 0.24.0) Status: Patch Available (was: Open) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246129#comment-13246129 ] Hadoop QA commented on HDFS-3102: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521291/HDFS-3102.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2181//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2181//console This message is automatically generated. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3194) Continuous block scanning at DN side
Continuous block scanning at DN side Key: HDFS-3194 URL: https://issues.apache.org/jira/browse/HDFS-3194 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.3 Reporter: suja s Priority: Minor Fix For: 1.0.3 Block scanning interval by default should be taken as 21 days(3 weeks) and each block scanning should happen once in 21 days. Here the block is being scanned continuosly. 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246143#comment-13246143 ] Hadoop QA commented on HDFS-3084: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521282/hdfs-3084.txt against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2182//console This message is automatically generated. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3175) When the disk space is available back,Namenode resource monitor can automatically take off safemode.
[ https://issues.apache.org/jira/browse/HDFS-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246150#comment-13246150 ] liaowenrui commented on HDFS-3175: -- hi Aaron, I think that automatically leave safemode mechanism is better than manual.and It should be optimized to automatical approach. Is it right? When the disk space is available back,Namenode resource monitor can automatically take off safemode. Key: HDFS-3175 URL: https://issues.apache.org/jira/browse/HDFS-3175 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.24.0, 2.0.0 Reporter: liaowenrui Attachments: HDFS-3175.patch, HDFS-3175.patch, HDFS-3175.patch, testcase -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3175) When the disk space is available back,Namenode resource monitor can automatically take off safemode.
[ https://issues.apache.org/jira/browse/HDFS-3175?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246164#comment-13246164 ] liaowenrui commented on HDFS-3175: -- Hi Aaron, I have seen your opinion in other issue HDFS-2914 https://issues.apache.org/jira/browse/HDFS-2914?focusedCommentId=13203135page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13203135 This can solve that issue as well. When the disk space is available back,Namenode resource monitor can automatically take off safemode. Key: HDFS-3175 URL: https://issues.apache.org/jira/browse/HDFS-3175 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Affects Versions: 0.24.0, 2.0.0 Reporter: liaowenrui Attachments: HDFS-3175.patch, HDFS-3175.patch, HDFS-3175.patch, testcase -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2964) No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated HDFS-2964: --- Attachment: HDFS-2964.patch Added a warning message in FsVolumeImpl.java where it reads the latest capacity. I m creating the patch first time so not sure its the correct place to add the warning message. No notice in any logs if dfs.datanode.du.reserved is greater than available disk space -- Key: HDFS-2964 URL: https://issues.apache.org/jira/browse/HDFS-2964 Project: Hadoop HDFS Issue Type: Bug Components: data-node, name-node Affects Versions: 0.20.2 Reporter: Robert J Berger Priority: Minor Attachments: HDFS-2964.patch We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that could only be replicated to 0 nodes, instead of 1. There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the reserverd space might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster. It seems that there should be some warning or error in the logs that say that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2964) No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246193#comment-13246193 ] Harsh J commented on HDFS-2964: --- Hello Madhukara, Please attach a proper diff that carries only the changes you have done. Your current diff is a total rewrite of file. Ensure you use unix newlines in your editor/save-actions/etc., in case that is what led to this. No notice in any logs if dfs.datanode.du.reserved is greater than available disk space -- Key: HDFS-2964 URL: https://issues.apache.org/jira/browse/HDFS-2964 Project: Hadoop HDFS Issue Type: Bug Components: data-node, name-node Affects Versions: 0.20.2 Reporter: Robert J Berger Priority: Minor Attachments: HDFS-2964.patch We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that could only be replicated to 0 nodes, instead of 1. There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the reserverd space might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster. It seems that there should be some warning or error in the logs that say that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246199#comment-13246199 ] Harsh J commented on HDFS-3194: --- org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner does not exist in branch-1. It is a branch-2 feature (attached to Federation). What version are you really using and reporting for? Please fix the version field to reflect the right version… Continuous block scanning at DN side Key: HDFS-3194 URL: https://issues.apache.org/jira/browse/HDFS-3194 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.3 Reporter: suja s Priority: Minor Fix For: 1.0.3 Block scanning interval by default should be taken as 21 days(3 weeks) and each block scanning should happen once in 21 days. Here the block is being scanned continuosly. 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2964) No notice in any logs if dfs.datanode.du.reserved is greater than available disk space
[ https://issues.apache.org/jira/browse/HDFS-2964?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] madhukara phatak updated HDFS-2964: --- Attachment: HDFS-2964-1.patch Fixed the patch file to have only changed code and converted new lines into UNIX No notice in any logs if dfs.datanode.du.reserved is greater than available disk space -- Key: HDFS-2964 URL: https://issues.apache.org/jira/browse/HDFS-2964 Project: Hadoop HDFS Issue Type: Bug Components: data-node, name-node Affects Versions: 0.20.2 Reporter: Robert J Berger Priority: Minor Attachments: HDFS-2964-1.patch, HDFS-2964.patch We spent a long time tracking down why a test hdfs cluster seemed to be running fine, but would not allow the mapred system to come up complaining that could only be replicated to 0 nodes, instead of 1. There were no namenode or datanode errors in any of the logs. hadoop fsck said everything was good. At first glance dfsadmin -report looked good. It wasn't until I realized that there was 0 Capacity available that we poked around and found https://groups.google.com/a/cloudera.org/group/scm-users/msg/a4252d6623adbc2d which mentioned that the reserverd space might be greater than the disk space available. And we did find that our dfs.datanode.du.reserved was indeed higher than our actual since we were only testing a small cluster. It seems that there should be some warning or error in the logs that say that. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3176) JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
[ https://issues.apache.org/jira/browse/HDFS-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246209#comment-13246209 ] Hudson commented on HDFS-3176: -- Integrated in Hadoop-Hdfs-0.23-Build #218 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/]) svn merge -c 1309114 from trunk for HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil. (Revision 1309115) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309115 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own. --- Key: HDFS-3176 URL: https://issues.apache.org/jira/browse/HDFS-3176 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 1.0.1 Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 1.1.0, 0.23.3, 2.0.0, 3.0.0 Attachments: hdfs-3176-branch-1.patch, hdfs-3176.patch Currently JsonUtil used by webhdfs parses MD5MD5CRC32FileChecksum binary bytes on its own and contructs a MD5MD5CRC32FileChecksum. It should instead call MD5MD5CRC32FileChecksum.readFields(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3166) Hftp connections do not have a timeout
[ https://issues.apache.org/jira/browse/HDFS-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246210#comment-13246210 ] Hudson commented on HDFS-3166: -- Integrated in Hadoop-Hdfs-0.23-Build #218 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/218/]) svn merge -c 1309103 from trunk for HDFS-3166. Add timeout to Hftp connections. (Revision 1309106) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309106 Files : * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/URLUtils.java * /hadoop/common/branches/branch-0.23/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpURLTimeouts.java Hftp connections do not have a timeout -- Key: HDFS-3166 URL: https://issues.apache.org/jira/browse/HDFS-3166 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.2, 0.23.3, 2.0.0, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: HADOOP-8221.branch-1.patch, HADOOP-8221.patch, HADOOP-8221.patch, HDFS-3166.patch, HDFS-3166.patch Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket. This may also affect WebHdfs, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3176) JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
[ https://issues.apache.org/jira/browse/HDFS-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246230#comment-13246230 ] Hudson commented on HDFS-3176: -- Integrated in Hadoop-Hdfs-trunk #1005 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/]) HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . Contributed by Kihwal Lee (Revision 1309114) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309114 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own. --- Key: HDFS-3176 URL: https://issues.apache.org/jira/browse/HDFS-3176 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 1.0.1 Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 1.1.0, 0.23.3, 2.0.0, 3.0.0 Attachments: hdfs-3176-branch-1.patch, hdfs-3176.patch Currently JsonUtil used by webhdfs parses MD5MD5CRC32FileChecksum binary bytes on its own and contructs a MD5MD5CRC32FileChecksum. It should instead call MD5MD5CRC32FileChecksum.readFields(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3000) Add a public API for setting quotas
[ https://issues.apache.org/jira/browse/HDFS-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246232#comment-13246232 ] Hudson commented on HDFS-3000: -- Integrated in Hadoop-Hdfs-trunk #1005 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/]) HDFS-3000. Add a public API for setting quotas. Contributed by Aaron T. Myers. (Revision 1309227) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309227 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHdfsAdmin.java Add a public API for setting quotas --- Key: HDFS-3000 URL: https://issues.apache.org/jira/browse/HDFS-3000 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch Currently one can set the quota of a file or directory from the command line, but if a user wants to set it programmatically, they need to use DistributedFileSystem, which is annotated InterfaceAudience.Private. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246231#comment-13246231 ] Hudson commented on HDFS-3168: -- Integrated in Hadoop-Hdfs-trunk #1005 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/]) HDFS-3168. Remove unnecessary throw IOException and change fields to final in FSNamesystem and BlockManager. (Revision 1309218) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309218 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3187) Upgrade guava to 11.0.2
[
https://issues.apache.org/jira/browse/HDFS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246233#comment-13246233
]
Hudson commented on HDFS-3187:
--
Integrated in Hadoop-Hdfs-trunk #1005 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/])
HDFS-3187. Upgrade guava to 11.0.2. Contributed by Todd Lipcon. (Revision
1309181)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309181
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSImageTestUtil.java
* /hadoop/common/trunk/hadoop-project/pom.xml
Upgrade guava to 11.0.2
---
Key: HDFS-3187
URL: https://issues.apache.org/jira/browse/HDFS-3187
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Fix For: 2.0.0
Attachments: hdfs-3187.txt
Guava r11 includes some nice features which we'd like to use in the
implementation of HDFS-3077. In particular,
{{MoreExecutors.listeningDecorator}} allows a normal {{ExecutorService}} to
be turned into a {{ListeningExecutorService}}, so that tasks can be submitted
to it and then wrapped as {{ListenableFuture}}s.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3166) Hftp connections do not have a timeout
[ https://issues.apache.org/jira/browse/HDFS-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246234#comment-13246234 ] Hudson commented on HDFS-3166: -- Integrated in Hadoop-Hdfs-trunk #1005 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1005/]) HDFS-3166. Add timeout to Hftp connections. Contributed by Daryn Sharp (Revision 1309103) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309103 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/URLUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpURLTimeouts.java Hftp connections do not have a timeout -- Key: HDFS-3166 URL: https://issues.apache.org/jira/browse/HDFS-3166 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.2, 0.23.3, 2.0.0, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: HADOOP-8221.branch-1.patch, HADOOP-8221.patch, HADOOP-8221.patch, HDFS-3166.patch, HDFS-3166.patch Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket. This may also affect WebHdfs, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-1362) Provide volume management functionality for DataNode
[ https://issues.apache.org/jira/browse/HDFS-1362?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246251#comment-13246251 ] Uma Maheswara Rao G commented on HDFS-1362: --- Any update on this? - Thanks Provide volume management functionality for DataNode Key: HDFS-1362 URL: https://issues.apache.org/jira/browse/HDFS-1362 Project: Hadoop HDFS Issue Type: New Feature Components: data-node Affects Versions: 0.23.0 Reporter: Wang Xu Assignee: Wang Xu Fix For: 0.24.0 Attachments: DataNode Volume Refreshment in HDFS-1362.pdf, HDFS-1362.4_w7001.txt, HDFS-1362.5.patch, HDFS-1362.6.patch, HDFS-1362.7.patch, HDFS-1362.8.patch, HDFS-1362.txt, Provide_volume_management_for_DN_v1.pdf The current management unit in Hadoop is a node, i.e. if a node failed, it will be kicked out and all the data on the node will be replicated. As almost all SATA controller support hotplug, we add a new command line interface to datanode, thus it can list, add or remove a volume online, which means we can change a disk without node decommission. Moreover, if the failed disk still readable and the node has enouth space, it can migrate data on the disks to other disks in the same node. A more detailed design document will be attached. The original version in our lab is implemented against 0.20 datanode directly, and is it better to implemented it in contrib? Or any other suggestion? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3122) Block recovery with closeFile flag true can race with blockReport. Due to this blocks are getting marked as corrupt.
[
https://issues.apache.org/jira/browse/HDFS-3122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246256#comment-13246256
]
amith commented on HDFS-3122:
-
Hi Folks
I thought of some sample solution for this problem please give your inputs to
the same.
When any blockrecovery happen, commitBlockSynchronization is called we can
store the old and new generation stamp for this block in a map which is like
historyMap = new HashMapString, ArrayListLong(). Here ArrayList is of size
2 which contain old and new GenerationStamp.
For every recovery this map is updated with the block and Generation Stamps.
Consider the scenario when BlockReport has arrived @ NN and delayed.
Now if the any BlockRecovery completed (historyMap will have the entry of old
and new Generation Stamps).
Now the Blockreport processing started. Here
{code}
case RWR:
if (!storedBlock.isComplete()) {
return null; // not corrupt
} else if (storedBlock.getGenerationStamp() != iblk.getGenerationStamp()
!historyMap.get(iblk.getBlockId()).get(0) !=
iblk.getGenerationStamp() )) {
return new BlockToMarkCorrupt(storedBlock,
reported + reportedState + replica with genstamp +
iblk.getGenerationStamp() + does not match COMPLETE block's +
genstamp in block map + storedBlock.getGenerationStamp());
} else { // COMPLETE block, same genstamp
{code}
Here we are checking like
if (Block GenerationStamp from BlockMap != BlockReport's Block GenerationStamp
and (blockGenerationStamp is newly changed due to recovery then check the)
old GenerationStamp is not equal BlockReports Block GenerationStamp) then { //
mark block as corrupt the block }
Map is populated in CommitBlockSynchronization and cleared when BlockReport is
processed for this block with new generationstamp
Block recovery with closeFile flag true can race with blockReport. Due to
this blocks are getting marked as corrupt.
Key: HDFS-3122
URL: https://issues.apache.org/jira/browse/HDFS-3122
Project: Hadoop HDFS
Issue Type: Bug
Components: data-node, name-node
Affects Versions: 0.23.0, 0.24.0
Reporter: Uma Maheswara Rao G
Assignee: Uma Maheswara Rao G
Priority: Critical
Attachments: blockCorrupt.txt
*Block Report* can *race* with *Block Recovery* with closeFile flag true.
Block report generated just before block recovery at DN side and due to N/W
problems, block report got delayed to NN.
After this, recovery success and generation stamp modifies to new one.
And primary DN invokes the commitBlockSynchronization and block got updated
in NN side. Also block got marked as complete, since the closeFile flag was
true. Updated with new genstamp.
Now blockReport started processing at NN side. This particular block from RBW
(when it generated the BR at DN), and file was completed at NN side.
Finally block will be marked as corrupt because of genstamp mismatch.
{code}
case RWR:
if (!storedBlock.isComplete()) {
return null; // not corrupt
} else if (storedBlock.getGenerationStamp() !=
iblk.getGenerationStamp()) {
return new BlockToMarkCorrupt(storedBlock,
reported + reportedState + replica with genstamp +
iblk.getGenerationStamp() + does not match COMPLETE block's +
genstamp in block map + storedBlock.getGenerationStamp());
} else { // COMPLETE block, same genstamp
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246268#comment-13246268 ] Daryn Sharp commented on HDFS-3194: --- I've seen this behavior in branch-2 branch-3. Last year, I think Matt Foley fixed this or a similar issue with excessive block scanning. This may be a regression. Continuous block scanning at DN side Key: HDFS-3194 URL: https://issues.apache.org/jira/browse/HDFS-3194 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.0.3 Reporter: suja s Priority: Minor Fix For: 1.0.3 Block scanning interval by default should be taken as 21 days(3 weeks) and each block scanning should happen once in 21 days. Here the block is being scanned continuosly. 2012-04-03 10:44:47,056 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:02,064 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:17,071 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 2012-04-03 10:45:32,079 INFO org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3176) JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own.
[ https://issues.apache.org/jira/browse/HDFS-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246274#comment-13246274 ] Hudson commented on HDFS-3176: -- Integrated in Hadoop-Mapreduce-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/]) HDFS-3176. Use MD5MD5CRC32FileChecksum.readFields() in JsonUtil . Contributed by Kihwal Lee (Revision 1309114) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309114 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/JsonUtil.java JsonUtil should not parse the MD5MD5CRC32FileChecksum bytes on its own. --- Key: HDFS-3176 URL: https://issues.apache.org/jira/browse/HDFS-3176 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.0, 1.0.1 Reporter: Kihwal Lee Assignee: Kihwal Lee Fix For: 1.1.0, 0.23.3, 2.0.0, 3.0.0 Attachments: hdfs-3176-branch-1.patch, hdfs-3176.patch Currently JsonUtil used by webhdfs parses MD5MD5CRC32FileChecksum binary bytes on its own and contructs a MD5MD5CRC32FileChecksum. It should instead call MD5MD5CRC32FileChecksum.readFields(). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246275#comment-13246275 ] Hudson commented on HDFS-3168: -- Integrated in Hadoop-Mapreduce-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/]) HDFS-3168. Remove unnecessary throw IOException and change fields to final in FSNamesystem and BlockManager. (Revision 1309218) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309218 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/ha/EditLogTailer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManagerTestUtil.java Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3000) Add a public API for setting quotas
[ https://issues.apache.org/jira/browse/HDFS-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246276#comment-13246276 ] Hudson commented on HDFS-3000: -- Integrated in Hadoop-Mapreduce-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/]) HDFS-3000. Add a public API for setting quotas. Contributed by Aaron T. Myers. (Revision 1309227) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309227 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/client/HdfsAdmin.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHdfsAdmin.java Add a public API for setting quotas --- Key: HDFS-3000 URL: https://issues.apache.org/jira/browse/HDFS-3000 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch Currently one can set the quota of a file or directory from the command line, but if a user wants to set it programmatically, they need to use DistributedFileSystem, which is annotated InterfaceAudience.Private. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3166) Hftp connections do not have a timeout
[ https://issues.apache.org/jira/browse/HDFS-3166?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246278#comment-13246278 ] Hudson commented on HDFS-3166: -- Integrated in Hadoop-Mapreduce-trunk #1040 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/]) HDFS-3166. Add timeout to Hftp connections. Contributed by Daryn Sharp (Revision 1309103) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309103 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/HsftpFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/DelegationTokenFetcher.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/URLUtils.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestHftpURLTimeouts.java Hftp connections do not have a timeout -- Key: HDFS-3166 URL: https://issues.apache.org/jira/browse/HDFS-3166 Project: Hadoop HDFS Issue Type: Bug Components: hdfs client Affects Versions: 0.23.2, 0.23.3, 2.0.0, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Critical Fix For: 0.23.3, 2.0.0, 3.0.0 Attachments: HADOOP-8221.branch-1.patch, HADOOP-8221.patch, HADOOP-8221.patch, HDFS-3166.patch, HDFS-3166.patch Hftp connections do not have read timeouts. This leads to indefinitely hung sockets when there is a network outage during which time the remote host closed the socket. This may also affect WebHdfs, etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3187) Upgrade guava to 11.0.2
[
https://issues.apache.org/jira/browse/HDFS-3187?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246277#comment-13246277
]
Hudson commented on HDFS-3187:
--
Integrated in Hadoop-Mapreduce-trunk #1040 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1040/])
HDFS-3187. Upgrade guava to 11.0.2. Contributed by Todd Lipcon. (Revision
1309181)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309181
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/FSImageTestUtil.java
* /hadoop/common/trunk/hadoop-project/pom.xml
Upgrade guava to 11.0.2
---
Key: HDFS-3187
URL: https://issues.apache.org/jira/browse/HDFS-3187
Project: Hadoop HDFS
Issue Type: Sub-task
Components: build
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Minor
Fix For: 2.0.0
Attachments: hdfs-3187.txt
Guava r11 includes some nice features which we'd like to use in the
implementation of HDFS-3077. In particular,
{{MoreExecutors.listeningDecorator}} allows a normal {{ExecutorService}} to
be turned into a {{ListeningExecutorService}}, so that tasks can be submitted
to it and then wrapped as {{ListenableFuture}}s.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246316#comment-13246316 ] Daryn Sharp commented on HDFS-3084: --- I haven't been following all the fencing jiras, so qq, are tokens and/or their services involved? FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3109) Remove hsqldb exclusions from pom.xml
[ https://issues.apache.org/jira/browse/HDFS-3109?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Thomas Graves updated HDFS-3109: Target Version/s: 0.23.3 (was: 0.23.2) Remove hsqldb exclusions from pom.xml - Key: HDFS-3109 URL: https://issues.apache.org/jira/browse/HDFS-3109 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.2 Reporter: Ravi Prakash Assignee: Ravi Prakash Attachments: HDFS-3109.patch Related to MAPREDUCE-3621 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3195) Compilation error in FSNamesystem
Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Priority: Blocker Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246381#comment-13246381 ] Jason Lowe commented on HDFS-3195: -- Appears to have been caused by HDFS-3168 per [this comment|https://issues.apache.org/jira/browse/HDFS-3168?focusedCommentId=13245996page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13245996]. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Priority: Blocker Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246385#comment-13246385 ] Flavio Junqueira commented on HDFS-3092: bq. Is there a way to turn off the striping even if Quorom size (Q) is less than Ensemble size (E)? We like the idea that each Journal file contains ALL entries. Our default config: Q is 2 and set of JDs is 3 (roughly equivalent to E). Right now the write set is the same as the ack set, since we haven't had the use case you suggest so far. We have anticipated the need of different ways of scheduling reads and writes, and it is fairly simple to make it cover your use case. Ivan had actually suggested this separation between ack sets and write sets previously to deal with slow disks. I have tried to reflect this discussion in BOOKKEEPER-208 if you're interested. Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: MultipleSharedJournals.pdf, MultipleSharedJournals.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins reassigned HDFS-3195: - Assignee: Eli Collins Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3195: -- Attachment: hdfs-3195.txt Patch attached. The throws clause was removed in HDFS-2301 and it looks like HDFS-3168 re-introduced it, which is kind of troublesome. I'll ping Nicholas on HDFS-3168 to double check that he didn't accidentally revert other changes. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] amith updated HDFS-3165: Status: Open (was: Patch Available) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.1 Environment: HDFS Balancer Reporter: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] amith updated HDFS-3165: Attachment: HDFS-3165.patch HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.1 Environment: HDFS Balancer Reporter: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned HDFS-3195: Assignee: Tsz Wo (Nicholas), SZE (was: Eli Collins) Nicholas, could you take a look at this? I think HDFS-3168 needs to be backed out of branch-2 or HDFS-2564 should be pulled into branch-2 to fix the build. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Tsz Wo (Nicholas), SZE Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Lowe reassigned HDFS-3195: Assignee: Eli Collins (was: Tsz Wo (Nicholas), SZE) Whoops, comment race, sorry back to Eli. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246397#comment-13246397 ] Eli Collins commented on HDFS-3195: --- Actually, it looks like HDFS-3168 removed the throws from reassignLease, which should have been done when HDFS-2301 (part of HA) was merged to trunk. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3165) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh
[ https://issues.apache.org/jira/browse/HDFS-3165?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246396#comment-13246396 ] amith commented on HDFS-3165: - Sorry for the delay in providing the correct patch. please accept the new patch which is attached newly :) HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh Key: HDFS-3165 URL: https://issues.apache.org/jira/browse/HDFS-3165 Project: Hadoop HDFS Issue Type: Bug Components: balancer Affects Versions: 1.0.1 Environment: HDFS Balancer Reporter: amith Priority: Minor Attachments: HDFS-3165.patch, HDFS-3165.patch Original Estimate: 1m Remaining Estimate: 1m HDFS Balancer scripts are refering to wrong path of hadoop-daemon.sh HOST-xx-xx-xx-xx:/home/amith/V1R2/namenode1/sbin # ./start-balancer.sh ./start-balancer.sh: line 27: /home/amith/V1R2/namenode1/bin/hadoop-daemon.sh: No such file or directory currently hadoop-daemon.sh script is in sbin and not bin. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246419#comment-13246419 ] Aaron T. Myers commented on HDFS-3102: -- The test failure appears to be unrelated. The test passes locally for me, and failed on the Jenkins slave with a java.io.IOException: Too many open files; error. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246422#comment-13246422 ] Eli Collins commented on HDFS-3195: --- ATM had a better idea, fixing this by merging HDFS-2564 to branch-2, I'll do that and close that out when I've committed and verified the build is fixed. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2564) Cleanup unnecessary exceptions thrown and unnecessary casts
[ https://issues.apache.org/jira/browse/HDFS-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2564: -- Target Version/s: (was: 0.23.1) Affects Version/s: (was: 0.24.0) Fix Version/s: (was: 3.0.0) 2.0.0 I've merged this to branch-2 Cleanup unnecessary exceptions thrown and unnecessary casts --- Key: HDFS-2564 URL: https://issues.apache.org/jira/browse/HDFS-2564 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client, name-node Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 2.0.0 Attachments: HDFS-2564-1.txt, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch Cleaning up some of the java files with unnecessary exceptions and casts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins resolved HDFS-3195. --- Resolution: Fixed I've merged HDFS-2564 to branch-2, which fixes this issue. Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246457#comment-13246457 ] Todd Lipcon commented on HDFS-3084: --- No, the scripts and failover controllers use keytab-based or straight user credentials. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2564) Cleanup unnecessary exceptions thrown and unnecessary casts
[ https://issues.apache.org/jira/browse/HDFS-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246458#comment-13246458 ] Hudson commented on HDFS-2564: -- Integrated in Hadoop-Hdfs-trunk-Commit #2069 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2069/]) Updated CHANGES.txt to reflect HDFS-2564 merge (Revision 1309488) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309488 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Cleanup unnecessary exceptions thrown and unnecessary casts --- Key: HDFS-2564 URL: https://issues.apache.org/jira/browse/HDFS-2564 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client, name-node Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 2.0.0 Attachments: HDFS-2564-1.txt, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch Cleaning up some of the java files with unnecessary exceptions and casts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2564) Cleanup unnecessary exceptions thrown and unnecessary casts
[ https://issues.apache.org/jira/browse/HDFS-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246460#comment-13246460 ] Hudson commented on HDFS-2564: -- Integrated in Hadoop-Common-trunk-Commit #1994 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1994/]) Updated CHANGES.txt to reflect HDFS-2564 merge (Revision 1309488) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309488 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Cleanup unnecessary exceptions thrown and unnecessary casts --- Key: HDFS-2564 URL: https://issues.apache.org/jira/browse/HDFS-2564 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client, name-node Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 2.0.0 Attachments: HDFS-2564-1.txt, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch Cleaning up some of the java files with unnecessary exceptions and casts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3000) Add a public API for setting quotas
[ https://issues.apache.org/jira/browse/HDFS-3000?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3000: - Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've just committed this to trunk and branch-2. Thanks a lot for the reviews. Add a public API for setting quotas --- Key: HDFS-3000 URL: https://issues.apache.org/jira/browse/HDFS-3000 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs client Affects Versions: 2.0.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 2.0.0 Attachments: HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch, HDFS-3000.patch Currently one can set the quota of a file or directory from the command line, but if a user wants to set it programmatically, they need to use DistributedFileSystem, which is annotated InterfaceAudience.Private. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3050) rework OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-3050: -- Target Version/s: 2.0.0 rework OEV to share more code with the NameNode --- Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch, HDFS-3050.014.patch, HDFS-3050.015.patch, HDFS-3050.016.patch, HDFS-3050.017.patch, HDFS-3050.018.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes. We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file. Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246501#comment-13246501 ] Eli Collins commented on HDFS-3102: --- Two minor comments, otherwise looks excellent - Since initializeSharedEdits swallows most IOEs and returns a boolean I'd update the throws javadoc to indicate an IOE isn't throw for most error cases. Or perhaps swallow the IOE from unlockAll and return false tehre as well - Good to add a test that the force option works, specifically w/o specifying it we don't blow away your existing shared edits dirs. The code obviously looks correct but this would be a bad bug to introduce so worth checking Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2564) Cleanup unnecessary exceptions thrown and unnecessary casts
[ https://issues.apache.org/jira/browse/HDFS-2564?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246502#comment-13246502 ] Hudson commented on HDFS-2564: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2006 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2006/]) Updated CHANGES.txt to reflect HDFS-2564 merge (Revision 1309488) Result = SUCCESS eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309488 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt Cleanup unnecessary exceptions thrown and unnecessary casts --- Key: HDFS-2564 URL: https://issues.apache.org/jira/browse/HDFS-2564 Project: Hadoop HDFS Issue Type: Improvement Components: data-node, hdfs client, name-node Reporter: Hari Mankude Assignee: Hari Mankude Priority: Minor Fix For: 2.0.0 Attachments: HDFS-2564-1.txt, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch, hadoop-2564.trunk.patch Cleaning up some of the java files with unnecessary exceptions and casts. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3102: - Attachment: HDFS-3102.patch Here's an updated patch which incorporates Eli's feedback. bq. Since initializeSharedEdits swallows most IOEs and returns a boolean I'd update the throws javadoc to indicate an IOE isn't throw for most error cases. Or perhaps swallow the IOE from unlockAll and return false tehre as well Good thinking. I made it catch/log/return in the case of an IOE from unlockAll, and removed the throws declaration. bq. Good to add a test that the force option works, specifically w/o specifying it we don't blow away your existing shared edits dirs. The code obviously looks correct but this would be a bad bug to introduce so worth checking Note that the force option isn't actually exposed to the user - it's only used in testing. Regardless, you're right that it's useful to add a test that we won't overwrite directories without confirmation. I've add a little test to do so. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246542#comment-13246542 ] Eli Collins commented on HDFS-3102: --- +1 looks good Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3050) rework OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3050: --- Attachment: HDFS-3050.019.patch * fix error handling bug that could lead to open files getting leaked (theoretically) * suppress javadoc warnings resulting from com.sun.* API use rework OEV to share more code with the NameNode --- Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch, HDFS-3050.014.patch, HDFS-3050.015.patch, HDFS-3050.016.patch, HDFS-3050.017.patch, HDFS-3050.018.patch, HDFS-3050.019.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes. We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file. Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3195) Compilation error in FSNamesystem
[ https://issues.apache.org/jira/browse/HDFS-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246564#comment-13246564 ] Tsz Wo (Nicholas), SZE commented on HDFS-3195: -- Very sorry that I did not check branch-2 when merging HDFS-3168 last night and thanks for fixing it! Compilation error in FSNamesystem - Key: HDFS-3195 URL: https://issues.apache.org/jira/browse/HDFS-3195 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0 Reporter: Jason Lowe Assignee: Eli Collins Priority: Blocker Attachments: hdfs-3195.txt Build fails when trying to build branch-2: [ERROR] /hadoop/src/branch-2/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java:[2741,32] unreported exception java.io.IOException; must be caught or declared to be thrown -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246572#comment-13246572 ] Tsz Wo (Nicholas), SZE commented on HDFS-3168: -- Also, not a big deal, but in the future please get a +1 from a committer before committing a patch. Aaron, it is nothing to do with it. Any contributor could review code. It was a merging problem. Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon updated HDFS-3084: -- Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 2.0.0 (was: 0.23.3, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed to branch-2 and trunk, thanks for reviews. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Fix For: 2.0.0 Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246589#comment-13246589 ] Hudson commented on HDFS-3084: -- Integrated in Hadoop-Hdfs-trunk-Commit #2070 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2070/]) HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309526 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Fix For: 2.0.0 Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246591#comment-13246591 ] Hudson commented on HDFS-3084: -- Integrated in Hadoop-Common-trunk-Commit #1995 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1995/]) HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309526 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Fix For: 2.0.0 Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3084) FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port
[ https://issues.apache.org/jira/browse/HDFS-3084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246596#comment-13246596 ] Hudson commented on HDFS-3084: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2007/]) HDFS-3084. FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port. Contributed by Todd Lipcon. (Revision 1309526) Result = SUCCESS todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309526 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/tools/NNHAServiceTarget.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSHAAdminMiniCluster.java FenceMethod.tryFence() and ShellCommandFencer should pass namenodeId as well as host:port - Key: HDFS-3084 URL: https://issues.apache.org/jira/browse/HDFS-3084 Project: Hadoop HDFS Issue Type: Improvement Components: ha Affects Versions: 0.24.0, 0.23.3 Reporter: Philip Zeyliger Assignee: Todd Lipcon Fix For: 2.0.0 Attachments: hdfs-3084.txt The FenceMethod interface passes along the host:port of the NN that needs to be fenced. That's great for the common case. However, it's likely necessary to have extra configuration parameters for fencing, and these are typically keyed off the nameserviceId.namenodeId (if, for nothing else, consistency with all the other parameters that are keyed off of namespaceId.namenodeId). Obviously this can be backed out from the host:port, but it's inconvenient, and requires iterating through all the configs. The shell interface exhibits the same issue: host:port is great for most fencers, but if you need extra configs (like the host:port of the power supply unit), those are harder to pipe through without the namenodeId. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246600#comment-13246600 ] Todd Lipcon commented on HDFS-3168: --- bq. Aaron, it is nothing to do with it. Any contributor could review code. It was a merging problem. Is that the case? I have no opinion on this particular patch and whether a different reviewer might have seen the issue. But I thought you had to get a committer +1 to commit things... Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246610#comment-13246610 ] Tsz Wo (Nicholas), SZE commented on HDFS-3168: -- Could the committer be the same as the contributor? Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246624#comment-13246624 ] Hadoop QA commented on HDFS-3102: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521349/HDFS-3102.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2183//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2183//console This message is automatically generated. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3178: - Attachment: h3178_20120404_svn_mv.patch The file should be 3178_20120404_svn_mv.patch, not h3179_20120403.patch. Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3178: - Attachment: h3179_20120403.patch h3179_20120403.patch: - renames StateMachine to StateHandler - moves JournalListener to a standalone interface. (The patch did not include removing the old files so that it could be applied cleanly after svn_mv.sh.) Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3178: - Attachment: (was: h3179_20120403.patch) Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246651#comment-13246651 ] Todd Lipcon commented on HDFS-3168: --- By my understanding of our policies, the committer who provides the +1 has to be someone separate than the patch author. On branches I'm fine being lax here, since we need three +1s to merge a branch, but on trunk, I think it merits a discussion if there is disagreement on what our policies are. Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3178: - Attachment: h3178_20120404.patch h3178_20120404.patch: same patch for Jenkins. Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3178: - Status: Patch Available (was: Open) Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246657#comment-13246657 ] Tsz Wo (Nicholas), SZE commented on HDFS-3168: -- By my understanding of our policies, the committer who provides the +1 has to be someone separate than the patch author. ... I think this is not true. After a committer provided a patch and a non-committer reviewed it, the same committer could commit the patch. Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246661#comment-13246661 ] Hari Mankude commented on HDFS-3192: bq.Why? I think it's an advantage that the FC may die and come back, or that you may start the FCs after the NNs. Well, if FC restarts the health monitoring within the timeout period, then NN will not die. However, if FC is having a gc pause or is not restarting, then NN should die. This is the first level of protection where in if NN is healthy, it can stonith itself. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3196) Implement JournalListener for writing journal to local disk
Implement JournalListener for writing journal to local disk --- Key: HDFS-3196 URL: https://issues.apache.org/jira/browse/HDFS-3196 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246671#comment-13246671 ] Todd Lipcon commented on HDFS-3192: --- Why add multiple stonith paths, given we need external stonith anyway? It just adds to the complexity by increasing the number of scenarios we have to debug, etc. That is to say: if the ZKFC dies, then it will lose its lock, and the other node will stonith this one when it takes over. What's the benefit of having it abort itself at the same time? In fact, it seems to be detrimental, because if it stays up, the other node can do a graceful transitionToStandby() call rather than having to do something more drastic like a full abort. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3050) rework OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246675#comment-13246675 ] Hadoop QA commented on HDFS-3050: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521352/HDFS-3050.019.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. -1 javadoc. The javadoc tool appears to have generated 2 warning messages. -1 javac. The applied patch generated 20 javac compiler warnings (more than the trunk's current 14 warnings). +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.server.namenode.TestValidateConfigurationSettings +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2184//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2184//console This message is automatically generated. rework OEV to share more code with the NameNode --- Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch, HDFS-3050.014.patch, HDFS-3050.015.patch, HDFS-3050.016.patch, HDFS-3050.017.patch, HDFS-3050.018.patch, HDFS-3050.019.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes. We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file. Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246676#comment-13246676 ] Mingjie Lai commented on HDFS-2185: --- (still posting comments here since the design doc is attached here) @todd Thanks for adding the manual failover section 2.7 in the design doc. However I have some questions for what you described in 2.7.2: - HAAdmin makes an RPC failoverToYou() to the target ZKFC - target ZKFC sends an RPC concedeLock() to the currently active ZKFC. - the active sends a transitionToStandby() RPC to its local node IMO the chain of RPCs is quite complicated, not easy to debug and troubleshoot in operation. Because you're trying to resolve the 2 problems, auto and manual failover, at one place -- ZKFC. How about seperate the 2 cases: - add commands at haadmin to start/stop autofailover - stop-autofailover requests all ZKFC to exitElection - start-autofailover requests all ZKFC to enterElction - haadmin is responsible for handle manual failover (as current implementation) - admins can only perform manual failover when autofailover is stopped - can be used to specify one particular active NN Pros: - existing manual fo code can be kept mostly - although new RPC is added to ZKFC but we don't need them to talk to each other. the manual failover logic is all handled at client -- haadmin. - easier to extend to the case of multiple standby NNs cons: - administrator needs to explicitly start/stop autofailover, in addition to ZKFC process. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: auto-failover, ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Fix For: Auto failover (HDFS-3042) Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3102: - Target Version/s: 0.24.0, 2.0.0 (was: 2.0.0, 0.24.0) Issue Type: New Feature (was: Improvement) Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246687#comment-13246687 ] Hari Mankude commented on HDFS-3192: bq. Why add multiple stonith paths, given we need external stonith anyway? It just adds to the complexity by increasing the number of scenarios we have to debug, etc. I thought we are not going to have external stonith using special devices and that is mainly the reason why we are going through hoops to implement fencing in journal daemons. bq. That is to say: if the ZKFC dies, then it will lose its lock, and the other node will stonith this one when it takes over. What's the benefit of having it abort itself at the same time? In fact, it seems to be detrimental, because if it stays up, the other node can do a graceful transitionToStandby() call rather than having to do something more drastic like a full abort. I disagree about two items here. 1. Why is the behaviour different from what happens when zkfc loses the ephemeral node? Currently zkfc when it loses the ephemeral node will shutdown the active NN. Similarly if active NN does not hear from zkfc, it implies that zkfc is dead, going through gc pause essentially resulting in loss of ephemeral node. 2. If active NN loses quorum, it has to shutdown. There is no way to do a transitionToStandby() especially since the log is updated after NN metadata is updated and there is no way to roll back the last update. This is just one of the issues that we are aware of where a rollback would be necessary. There might be other situations where rollback is required. In fact, one of the most of the difficult APIs to implement correctly would be transitionToStandby() from active state. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3102: - Resolution: Fixed Fix Version/s: 2.0.0 Target Version/s: 0.24.0, 2.0.0 (was: 2.0.0, 0.24.0) Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks a lot for the reviews, Eli. I've just committed this to trunk and branch-2. Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 2.0.0 Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246696#comment-13246696 ] Hudson commented on HDFS-3102: -- Integrated in Hadoop-Hdfs-trunk-Commit #2072 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2072/]) HDFS-3102. Add CLI tool to initialize the shared-edits dir. Contributed by Aaron T. Myers. (Revision 1309580) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 2.0.0 Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246700#comment-13246700 ] Hudson commented on HDFS-3102: -- Integrated in Hadoop-Common-trunk-Commit #1997 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1997/]) HDFS-3102. Add CLI tool to initialize the shared-edits dir. Contributed by Aaron T. Myers. (Revision 1309580) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 2.0.0 Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3102) Add CLI tool to initialize the shared-edits dir
[ https://issues.apache.org/jira/browse/HDFS-3102?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246709#comment-13246709 ] Hudson commented on HDFS-3102: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2009 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2009/]) HDFS-3102. Add CLI tool to initialize the shared-edits dir. Contributed by Aaron T. Myers. (Revision 1309580) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1309580 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/common/HdfsServerConstants.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestBootstrapStandby.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java Add CLI tool to initialize the shared-edits dir --- Key: HDFS-3102 URL: https://issues.apache.org/jira/browse/HDFS-3102 Project: Hadoop HDFS Issue Type: New Feature Components: ha, name-node Affects Versions: 0.24.0, 2.0.0 Reporter: Todd Lipcon Assignee: Aaron T. Myers Fix For: 2.0.0 Attachments: HDFS-3102.patch, HDFS-3102.patch, HDFS-3102.patch Currently in order to make a non-HA NN HA, you need to initialize the shared edits dir. This can be done manually by cping directories around. It would be preferable to add a namenode -initializeSharedEdits command to achieve this same effect. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246718#comment-13246718 ] Todd Lipcon commented on HDFS-3192: --- bq. I thought we are not going to have external stonith using special devices and that is mainly the reason why we are going through hoops to implement fencing in journal daemons. In the current design, which uses a filer, we *require* external stonith devices. There is no correct way of doing it without either stonith or storage fencing. The proposal with the journal-daemon based fencing is essentailly the same as storage fencing - just that we do it with our own software storage instead of a NAS/SAN. bq. Why is the behaviour different from what happens when zkfc loses the ephemeral node? Currently zkfc when it loses the ephemeral node will shutdown the active NN No, it doesn't - it will transition it to standby. But, as I commented elsewhere, this is redundant, because the _new_ active is actually going to fence it anyway before taking over. bq. Similarly if active NN does not hear from zkfc, it implies that zkfc is dead, going through gc pause essentially resulting in loss of ephemeral node. But this can reduce uptime. For example, imagine an administrator accidentally changes the ACL on zookeeper. This causes both ZKFCs to get an authentication error and crash at the same time. With your design, both NNs will then commit suicide. With the existing implementation, the system will continue to run in its existing state -- i.e no new failovers will occur, but whoever is active will remain active. bq. If active NN loses quorum, it has to shutdown Yes, it has to shut down _before_ it does any edits, or it has to be fenced by the next active. Notification of session loss is asynchronous. The same is true of your proposal. In either case it can take arbitrarily long before it notices that it should not be active. So we still require that the new active fence it before it becomes active. So, this proposal doesn't solve any problems. bq. In fact, one of the most of the difficult APIs to implement correctly would be transitionToStandby() from active state. We already have that implemented. It syncs any existing edits, and then stops allowing new ones. We allow failover from one node to another without aborting, so long as it's graceful. This is perfectly correct. If we need to do a non-graceful failover, we fence the node by STONITH or by disallowing further access to the edit logs (which indirectly causes the node to abort, since logSync() fails). It seems you're trying to solve problems we've already solved. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[
https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246720#comment-13246720
]
Aaron T. Myers commented on HDFS-3168:
--
bq. Aaron, it is nothing to do with it. Any contributor could review code. It
was a merging problem.
I definitely didn't mean to imply that this particular compilation issue would
have been caught by a different reviewer. I'm confident I would not have caught
this particular issue myself. :)
bq. I think this is not true. After a committer provided a patch and a
non-committer reviewed it, the same committer could commit the patch.
I think the bylaws are a little ambiguous on the subject. Per the bylaws:
{quote}
*Code Change*
A change made to a codebase of the project and committed by a committer. This
includes source code, documentation, website content, etc. Lazy consensus of
active committers, but with a minimum of one +1. The code can be committed
after the first +1, unless the code change represents a merge from a branch, in
which case three +1s are required.
{quote}
This would seem to imply that a review by a non-committer contributor is
non-binding. It does not, however, clear up the issue of whether or not a
committer can provide a +1 of their own patch.
FWIW, my understanding is the same as Todd's on this subject.
Clean up FSNamesystem and BlockManager
--
Key: HDFS-3168
URL: https://issues.apache.org/jira/browse/HDFS-3168
Project: Hadoop HDFS
Issue Type: Sub-task
Components: name-node
Reporter: Tsz Wo (Nicholas), SZE
Assignee: Tsz Wo (Nicholas), SZE
Fix For: 0.24.0, 0.23.3
Attachments: h3168_20120330.patch, h3168_20120402.patch,
h3168_20120403.patch
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3121) hdfs tests for HADOOP-8014
[ https://issues.apache.org/jira/browse/HDFS-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246724#comment-13246724 ] Suresh Srinivas commented on HDFS-3121: --- Comments: # Not very clear THis test is test the fix.. Instead of describing running into serialization problem etc. can you just describe what the test is. Also please move the description as class level javadoc. # minor: defaultBLockSize - defaultBlockSize # Please ensure line lengths are with in 80 chars limit # testGetDefaultBlockSize() indententation at try is not correct. I am also not clear about the comment createFile... What has createFile got to do with the test? # Please add a brief description of what the test is testing. # Please consider: When expecting an exception in tests, you can move fail() within try clause, after the method that you expect exception from. This avoids also return from catch blocks. hdfs tests for HADOOP-8014 -- Key: HDFS-3121 URL: https://issues.apache.org/jira/browse/HDFS-3121 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.23.2, 0.23.3 Reporter: John George Assignee: John George Attachments: hdfs-3121.patch, hdfs-3121.patch, hdfs-3121.patch This JIRA is to write tests for viewing quota using viewfs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246725#comment-13246725 ] Hadoop QA commented on HDFS-3178: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12521374/h3178_20120404.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 6 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2185//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2185//console This message is automatically generated. Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2185) HA: HDFS portion of ZK-based FailoverController
[ https://issues.apache.org/jira/browse/HDFS-2185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246729#comment-13246729 ] Todd Lipcon commented on HDFS-2185: --- Hi Mingjie. Thanks for taking a look. The idea for the chain of RPCs is from talking with some folks here who work on Hadoop deployment. Their opinion was the following: currently, most of the Hadoop client tools are too thick. For example, in the current manual failover implementation, the fencing is run on the admin client. This means that you have to run the haadmin command from a machine that has access to all of the necessary fencing scripts, key files, etc. That's a little bizarre -- you would expect to configure these kinds of things only on the central location, not on the client. So, we decided that it makes sense to push the management of the whole failover process into the FCs themselves, and just use a single RPC to kick off the whole failover process. This keeps the client thin. As for your proposed alternative, here are a few thoughts: bq. existing manual fo code can be kept mostly We actually share much of the code already. But, the problem with using the existing code exactly as is, is that the failover controllers always expect to have complete control over the system. If the state of the NNs changes underneath the ZKFC, then the state in ZK will become inconsistent with the actual state of the system, and it's very easy to get into split brain scenarios. So, the idea is that, when auto-failover is enabled, *all* decisions must be made by ZKFCs. That way we can make sure the ZK state doesn't get out of sync. bq. although new RPC is added to ZKFC but we don't need them to talk to each other. the manual failover logic is all handled at client – haadmin. As noted above I think this is a con, not a pro, because it requires configuring fencing scripts at the client, and likely requiring that the client have read-write access to ZK bq. easier to extend to the case of multiple standby NNs I think the extension path to multiple standby is actually equally easy with both approaches. The solution in the ZKFC-managed implementation is to add a new znode like PreferredActive and have nodes avoid becoming active unless they're listed as preferred. The target node of the failover can just set itself to be preferred before asking the other node to cede the lock. Some other advantages that I probably didn't explain well in the design doc: - this design is fault tolerant. If the target node crashes in the middle of the process, then the old active will automatically regain the active state after its rejoin timeout elapses. With a client-managed setup, a well-meaning admin may ^C the process in the middle and leave the system with no active at all. - no need to introduce disable/enable to auto-failover. Just having both nodes quit the election wouldn't work, since one would end up quitting before the other, causing a blip where an unnecessary (random) failover occurred. We could carefully orchestrate the order of quitting, so the active quits last, but I think it still gets complicated. HA: HDFS portion of ZK-based FailoverController --- Key: HDFS-2185 URL: https://issues.apache.org/jira/browse/HDFS-2185 Project: Hadoop HDFS Issue Type: Sub-task Components: auto-failover, ha Affects Versions: 0.24.0, 0.23.3 Reporter: Eli Collins Assignee: Todd Lipcon Fix For: Auto failover (HDFS-3042) Attachments: Failover_Controller.jpg, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, hdfs-2185.txt, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.pdf, zkfc-design.tex This jira is for a ZK-based FailoverController daemon. The FailoverController is a separate daemon from the NN that does the following: * Initiates leader election (via ZK) when necessary * Performs health monitoring (aka failure detection) * Performs fail-over (standby to active and active to standby transitions) * Heartbeats to ensure the liveness It should have the same/similar interface as the Linux HA RM to aid pluggability. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3178) Add states for journal synchronization in journal daemon
[ https://issues.apache.org/jira/browse/HDFS-3178?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246731#comment-13246731 ] Todd Lipcon commented on HDFS-3178: --- Hi Nicholas. Could you please add some javadoc to the state enum values explaining the purpose of each state, and what the transitions are between them? Or augment the design doc for HDFS-3092 with this state machine, and reference it from the code? Add states for journal synchronization in journal daemon Key: HDFS-3178 URL: https://issues.apache.org/jira/browse/HDFS-3178 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Attachments: h3178_20120403_svn_mv.patch, h3178_20120404.patch, h3178_20120404_svn_mv.patch, svn_mv.sh Journal in a new daemon has to be synchronized to the current transaction. It requires new states such as WaitingForRoll, Syncing and Synced. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3168) Clean up FSNamesystem and BlockManager
[ https://issues.apache.org/jira/browse/HDFS-3168?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246734#comment-13246734 ] Todd Lipcon commented on HDFS-3168: --- Let's ask the dev list. I'll start a thread. Clean up FSNamesystem and BlockManager -- Key: HDFS-3168 URL: https://issues.apache.org/jira/browse/HDFS-3168 Project: Hadoop HDFS Issue Type: Sub-task Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE Fix For: 0.24.0, 0.23.3 Attachments: h3168_20120330.patch, h3168_20120402.patch, h3168_20120403.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246752#comment-13246752 ] Hari Mankude commented on HDFS-3192: bq. bq. Why is the behaviour different from what happens when zkfc loses the ephemeral node? Currently zkfc when it loses the ephemeral node will shutdown the active NN bq. No, it doesn't - it will transition it to standby. But, as I commented elsewhere, this is redundant, because the new active is actually going to fence it anyway before taking over. Well this is incorrect behaviour. It does not handle the situation that I mentioned earlier about requiring rollbacks. The transition to standby will have result in old active having incorrect in-memory state. The only way around this is to shutdown the active. The reason is that as soon zkfc on NN1 has lost the ephemeral znode, it is possible that zkfc on NN2 has taken over the znode and NN2 has started fencing the journals. There is no way to gracefully coordinate this with NN1. This would result in NN1 getting quorum loss which in turn could leave the in-memory state in NN1 in an inconsistent shape. Do you agree again that in-memory state of NN1 is inconsistent with the editlogs? bq. Similarly if active NN does not hear from zkfc, it implies that zkfc is dead, going through gc pause essentially resulting in loss of ephemeral node. bq. But this can reduce uptime. For example, imagine an administrator accidentally changes the ACL on zookeeper. This causes both ZKFCs to get an authentication error and crash at the same time. With your design, both NNs will then commit suicide. With the existing implementation, the system will continue to run in its existing state – i.e no new failovers will occur, but whoever is active will remain active. Firstly, how often does some change ACLs in zookeeper? Secondly, why is ZKFC dying when this happens? ZKFC must be more robust than NN. NN is a resource that is controlled by ZKFC. We should make zkfc more robust to handle zookeeper acl changes if this is a common occurance. bq. If active NN loses quorum, it has to shutdown bq. Yes, it has to shut down before it does any edits, or it has to be fenced by the next active. Notification of session loss is asynchronous. The same is true of your proposal. In either case it can take arbitrarily long before it notices that it should not be active. So we still require that the new active fence it before it becomes active. So, this proposal doesn't solve any problems. My proposal was not meant to handle active NN losing quorum. My proposal is shutdown NN when ZKFC has died or is in a gc pause. My comment was with regards to earlier comment regarding doing a transitionToStandby(). Do you agree that active NN has invalid in-memory state and cannot go through transitionToStandby() when it loses quorum? There seems to be two solutions. 1. Implement rollback for various types of editlog entries and then do transitionToStandby() OR 2. Shutdown NN when it loses quorum Does this sound right? Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3197) Duplicate (and incorrect) class comments in a few tests
Duplicate (and incorrect) class comments in a few tests
---
Key: HDFS-3197
URL: https://issues.apache.org/jira/browse/HDFS-3197
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 0.24.0
Reporter: Aaron T. Myers
Priority: Trivial
Somewhat hilariously, TestFileCreationClient, TestDatanodeDeath,
TestReplaceDatanodeOnFailure, and TestDatanodeRegistration all have the
following class comment:
{noformat}
/**
* This class tests that a file need not be closed before its
* data can be read by another client.
*/
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3192) Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs
[ https://issues.apache.org/jira/browse/HDFS-3192?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246778#comment-13246778 ] Todd Lipcon commented on HDFS-3192: --- I think there is confusion here over the terminology loses quorum. I agree completely about the following: if any NN fails to sync its edit logs, it needs to abort. This is already what it does today - no changes necessary. If the edit log happens to be implemented using a quorum protocol (HDFS-3092 or HDFS-3077 for example), then that behavior should be maintained. The JournalManager implementation needs to throw an exception in response to logSync(). That will cause the NN to abort. That's all that's necessary for correctness - an NN won't ack success to a write unless it successfully syncs it, and will abort rather than rollback, since we have no rollback capability. In the above sense, loses quorum really means loses write access to the edit logs. If instead you're talking about loses quorum as loses its ZK session, then no abort is necessary, because it may still be able to write to its edits. So long as it's getting success back from editLog.logSync(), then the edits are being persisted. It is the responsibility of the next active to fence access to the shared edits. It may do so in one of two ways: 1) Edits fencing: ensure that the next write to the edits mechanism throws IOE. In the case of FileJournalManager on NAS, this is done via an RPC to the NAS system to fence the given export. 2) STONITH: ensure that the next write fails because power has been yanked from the machine. Alternatively, the new active may first try a graceful transition: 3) Gracefully ask the prior active to stop writing. The prior active flushes anything buffered, successfully syncs, and then enters standby mode. Notably, self-stonith upon losing the ZK lease is not an option, because it may take arbitrarily long before it notices. EG: 1) NN1 writing to edits log 2) ZKFC1 loses lease, but doesn't know about it yet 3) ZKFC2 gets lease 4) NN2 becomes active, starts writing logs 5) NN1 writes some edits. World explodes. 6) ZKFC1 gets asynchronous notification from ZK that it lots its session. Anything you do at this point is _too late_. Before step 4, NN2 must use a fencing mechanism. *Regardless* of whatever steps NN1 or ZKFC1 might take in step 6. Active NN should exit when it has not received a getServiceStatus() rpc from ZKFC for timeout secs -- Key: HDFS-3192 URL: https://issues.apache.org/jira/browse/HDFS-3192 Project: Hadoop HDFS Issue Type: Sub-task Components: ha, name-node Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3050) rework OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246784#comment-13246784 ] Colin Patrick McCabe commented on HDFS-3050: The TestValidateConfigurationSettings is another random jenkins failure (port in use). rework OEV to share more code with the NameNode --- Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch, HDFS-3050.014.patch, HDFS-3050.015.patch, HDFS-3050.016.patch, HDFS-3050.017.patch, HDFS-3050.018.patch, HDFS-3050.019.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes. We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file. Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3050) rework OEV to share more code with the NameNode
[ https://issues.apache.org/jira/browse/HDFS-3050?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3050: --- Attachment: HDFS-3050.020.patch * update OK_JAVADOC_WARNINGS rework OEV to share more code with the NameNode --- Key: HDFS-3050 URL: https://issues.apache.org/jira/browse/HDFS-3050 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor Attachments: HDFS-3050.006.patch, HDFS-3050.007.patch, HDFS-3050.008.patch, HDFS-3050.009.patch, HDFS-3050.010.patch, HDFS-3050.011.patch, HDFS-3050.012.patch, HDFS-3050.014.patch, HDFS-3050.015.patch, HDFS-3050.016.patch, HDFS-3050.017.patch, HDFS-3050.018.patch, HDFS-3050.019.patch, HDFS-3050.020.patch Current, OEV (the offline edits viewer) re-implements all of the opcode parsing logic found in the NameNode. This duplicated code creates a maintenance burden for us. OEV should be refactored to simply use the normal EditLog parsing code, rather than rolling its own. By using the existing FSEditLogLoader code to load edits in OEV, we can avoid having to update two places when the format changes. We should not put opcode checksums into the XML, because they are a serialization detail, not related to what the data is what we're storing. This will also make it possible to modify the XML file and translate this modified file back to a binary edits log file. Finally, this changes introduces --fix-txids. When OEV is passed this flag, it will close gaps in the transaction log by modifying the sequence numbers. This is useful if you want to modify the edit log XML (say, by removing a transaction), and transform the modified XML back into a valid binary edit log file. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2696) fuse-dfs build problems
[ https://issues.apache.org/jira/browse/HDFS-2696?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13246794#comment-13246794 ] Eli Collins commented on HDFS-2696: --- Hey Bruno, Can you generate one patch that applies against trunk so I can review? Thanks! fuse-dfs build problems --- Key: HDFS-2696 URL: https://issues.apache.org/jira/browse/HDFS-2696 Project: Hadoop HDFS Issue Type: Bug Components: contrib/fuse-dfs Affects Versions: 0.23.1 Environment: linux Reporter: Petru Dimulescu Labels: bigtop, build, maven, patch Fix For: 0.23.3 Attachments: HDFS-2696-plus.patch, dfsfuse-build.patch.zip fuse-dfs has some compilation problems on the 0.23 and 0.24(head) branches. Some details on how to compile fuse-dfs on 0.23 is here : http://mail-archives.apache.org/mod_mbox/hadoop-hdfs-user/201112.mbox/%3c4ee23195.7030...@gmail.com%3E and a wiki page follows. The attached patch removes some problems (an inexistnant ivy/library.properties, detecting jni.h location). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
