[jira] [Updated] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3049:
---
Target Version/s: 2.0.0
Affects Version/s: 0.23.0
Fix Version/s: (was: 0.24.0)
> During the normal loading NN startup process, fall back on a different
> EditLog if we see one that is corrupt
>
>
> Key: HDFS-3049
> URL: https://issues.apache.org/jira/browse/HDFS-3049
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Attachments: HDFS-3049.001.patch
>
>
> During the NameNode startup process, we load an image, and then apply edit
> logs to it until we believe that we have all the latest changes.
> Unfortunately, if there is an I/O error while reading any of these files, in
> most cases, we simply abort the startup process. We should try harder to
> locate a readable edit log and/or image file.
> *There are three main use cases for this feature:*
> 1. If the operating system does not honor fsync (usually due to a
> misconfiguration), a file may end up in an inconsistent state.
> 2. In certain older releases where we did not use fallocate() or similar to
> pre-reserve blocks, a disk full condition may cause a truncated log in one
> edit directory.
> 3. There may be a bug in HDFS which results in some of the data directories
> receiving corrupt data, but not all. This is the least likely use case.
> *Proposed changes to normal NN startup*
> * We should try a different FSImage if we can't load the first one we try.
> * We should examine other FSEditLogs if we can't load the first one(s) we try.
> * We should fail if we can't find EditLogs that would bring us up to what we
> believe is the latest transaction ID.
> Proposed changes to recovery mode NN startup:
> we should list out all the available storage directories and allow the
> operator to select which one he wants to use.
> Something like this:
> {code}
> Multiple storage directories found.
> 1. /foo/bar
> edits__curent__XYZ size:213421345 md5:2345345
> image size:213421345 md5:2345345
> 2. /foo/baz
> edits__curent__XYZ size:213421345 md5:2345345345
> image size:213421345 md5:2345345
> Which one would you like to use? (1/2)
> {code}
> As usual in recovery mode, we want to be flexible about error handling. In
> this case, this means that we should NOT fail if we can't find EditLogs that
> would bring us up to what we believe is the latest transaction ID.
> *Not addressed by this feature*
> This feature will not address the case where an attempt to access the
> NameNode name directory or directories hangs because of an I/O error. This
> may happen, for example, when trying to load an image from a hard-mounted NFS
> directory, when the NFS server has gone away. Just as now, the operator will
> have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3134: --- Target Version/s: 2.0.0 Fix Version/s: (was: 2.0.0) > harden edit log loader against malformed or malicious input > --- > > Key: HDFS-3134 > URL: https://issues.apache.org/jira/browse/HDFS-3134 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, > HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch > > > Currently, the edit log loader does not handle bad or malicious input > sensibly. > We can often cause OutOfMemory exceptions, null pointer exceptions, or other > unchecked exceptions to be thrown by feeding the edit log loader bad input. > In some environments, an out of memory error can cause the JVM process to be > terminated. > It's clear that we want these exceptions to be thrown as IOException instead > of as unchecked exceptions. We also want to avoid out of memory situations. > The main task here is to put a sensible upper limit on the lengths of arrays > and strings we allocate on command. The other task is to try to avoid > creating unchecked exceptions (by dereferencing potentially-NULL pointers, > for example). Instead, we should verify ahead of time and give a more > sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3280:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Committed to branch-2 (as noted above, this problem isn't present in trunk
since sync() was removed).
Thanks also to Matteo Bertozzi and Jon Hsieh for their help tracking this issue
down.
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253951#comment-13253951
]
Todd Lipcon commented on HDFS-3280:
---
bq. Ah, so this explains what you guys thought might be an interaction with
Nagle?
Yep, turned out to be much simpler :)
The patch failed on Hudson due to HDFS-3034 having removed the deprecated
method. I'll commit this based on Aaron's +1 and based on my manual stress
testing using HBase's HLog class, which uses this method.
No unit tests since it's hard to unit test for performance, and the hflush
equivalent is already tested by TestMultithreadedHflush
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253948#comment-13253948
]
Hadoop QA commented on HDFS-3280:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522653/hdfs-3280.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2277//console
This message is automatically generated.
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253946#comment-13253946
]
Aaron T. Myers commented on HDFS-3280:
--
+1 pending Jenkins.
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253945#comment-13253945
]
Hadoop QA commented on HDFS-3049:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522637/HDFS-3049.001.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.hdfs.server.namenode.TestFSEditLogLoader
org.apache.hadoop.hdfs.server.namenode.TestGenericJournalConf
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2276//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2276//console
This message is automatically generated.
> During the normal loading NN startup process, fall back on a different
> EditLog if we see one that is corrupt
>
>
> Key: HDFS-3049
> URL: https://issues.apache.org/jira/browse/HDFS-3049
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-3049.001.patch
>
>
> During the NameNode startup process, we load an image, and then apply edit
> logs to it until we believe that we have all the latest changes.
> Unfortunately, if there is an I/O error while reading any of these files, in
> most cases, we simply abort the startup process. We should try harder to
> locate a readable edit log and/or image file.
> *There are three main use cases for this feature:*
> 1. If the operating system does not honor fsync (usually due to a
> misconfiguration), a file may end up in an inconsistent state.
> 2. In certain older releases where we did not use fallocate() or similar to
> pre-reserve blocks, a disk full condition may cause a truncated log in one
> edit directory.
> 3. There may be a bug in HDFS which results in some of the data directories
> receiving corrupt data, but not all. This is the least likely use case.
> *Proposed changes to normal NN startup*
> * We should try a different FSImage if we can't load the first one we try.
> * We should examine other FSEditLogs if we can't load the first one(s) we try.
> * We should fail if we can't find EditLogs that would bring us up to what we
> believe is the latest transaction ID.
> Proposed changes to recovery mode NN startup:
> we should list out all the available storage directories and allow the
> operator to select which one he wants to use.
> Something like this:
> {code}
> Multiple storage directories found.
> 1. /foo/bar
> edits__curent__XYZ size:213421345 md5:2345345
> image size:213421345 md5:2345345
> 2. /foo/baz
> edits__curent__XYZ size:213421345 md5:2345345345
> image size:213421345 md5:2345345
> Which one would you like to use? (1/2)
> {code}
> As usual in recovery mode, we want to be flexible about error handling. In
> this case, this means that we should NOT fail if we can't find EditLogs that
> would bring us up to what we believe is the latest transaction ID.
> *Not addressed by this feature*
> This feature will not address the case where an attempt to access the
> NameNode name directory or directories hangs because of an I/O error. This
> may happen, for example, when trying to load an image from a hard-mounted NFS
> directory, when the NFS server has gone away. Just as now, the operator will
> have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3281) TestParallelRead times out with Clover enabled
TestParallelRead times out with Clover enabled -- Key: HDFS-3281 URL: https://issues.apache.org/jira/browse/HDFS-3281 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Henry Robinson Assignee: Henry Robinson Although it passes in a respectable time in pre-commit builds, when Clover is enabled TestParallelRead timesout (so runs > 15 minutes). I haven't been able to reproduce this on my development machines - the test runs in ~3 minutes and I don't think there's a factor of five difference in speed between this and the Jenkins box. I'm continuing to investigate the cause. If we can't find anything in the short term, we should temporarily disable this test to let the build go back to passing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253942#comment-13253942
]
Andrew Purtell commented on HDFS-3280:
--
Ah, so this explains what you guys thought might be an interaction with Nagle?
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3280:
--
Status: Patch Available (was: Open)
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253941#comment-13253941
]
Todd Lipcon commented on HDFS-3280:
---
Verified that this increased my benchmark performance by a factor of two.
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3280) DFSOutputStream.sync should not be synchronized
[
https://issues.apache.org/jira/browse/HDFS-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Todd Lipcon updated HDFS-3280:
--
Attachment: hdfs-3280.txt
Trivial patch should fix the issue. I'll verify with an HLog benchmark on a
cluster as well.
This is definitely correct since the method is just a one-line call to another
method which is unsynchronized.
> DFSOutputStream.sync should not be synchronized
> ---
>
> Key: HDFS-3280
> URL: https://issues.apache.org/jira/browse/HDFS-3280
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client
>Affects Versions: 2.0.0
>Reporter: Todd Lipcon
>Assignee: Todd Lipcon
>Priority: Critical
> Attachments: hdfs-3280.txt
>
>
> HDFS-895 added an optimization to make hflush() much faster by
> unsynchronizing it. But, we forgot to un-synchronize the deprecated
> {{sync()}} wrapper method. This makes the HBase WAL really slow on 0.23+
> since it doesn't take advantage of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3280) DFSOutputStream.sync should not be synchronized
DFSOutputStream.sync should not be synchronized
---
Key: HDFS-3280
URL: https://issues.apache.org/jira/browse/HDFS-3280
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs client
Affects Versions: 2.0.0
Reporter: Todd Lipcon
Assignee: Todd Lipcon
Priority: Critical
HDFS-895 added an optimization to make hflush() much faster by unsynchronizing
it. But, we forgot to un-synchronize the deprecated {{sync()}} wrapper method.
This makes the HBase WAL really slow on 0.23+ since it doesn't take advantage
of HDFS-895 anymore.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253937#comment-13253937 ] Hudson commented on HDFS-2708: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2088 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2088/]) HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253932#comment-13253932 ] Hudson commented on HDFS-2708: -- Integrated in Hadoop-Hdfs-trunk-Commit #2147 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2147/]) HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253933#comment-13253933 ] Hudson commented on HDFS-2708: -- Integrated in Hadoop-Common-trunk-Commit #2074 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2074/]) HDFS-2708. Stats for the # of blocks per DN. Contributed by Aaron T. Myers. (Revision 1326039) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326039 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameNodeMXBean.java > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2708: - Resolution: Fixed Fix Version/s: 2.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Thanks a lot for the review, Todd. I've just committed this to trunk and branch-2. > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Fix For: 2.0.0 > > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253923#comment-13253923 ] Hadoop QA commented on HDFS-2708: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522641/HDFS-2708.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2275//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2275//console This message is automatically generated. > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3049:
---
Status: Patch Available (was: Open)
> During the normal loading NN startup process, fall back on a different
> EditLog if we see one that is corrupt
>
>
> Key: HDFS-3049
> URL: https://issues.apache.org/jira/browse/HDFS-3049
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-3049.001.patch
>
>
> During the NameNode startup process, we load an image, and then apply edit
> logs to it until we believe that we have all the latest changes.
> Unfortunately, if there is an I/O error while reading any of these files, in
> most cases, we simply abort the startup process. We should try harder to
> locate a readable edit log and/or image file.
> *There are three main use cases for this feature:*
> 1. If the operating system does not honor fsync (usually due to a
> misconfiguration), a file may end up in an inconsistent state.
> 2. In certain older releases where we did not use fallocate() or similar to
> pre-reserve blocks, a disk full condition may cause a truncated log in one
> edit directory.
> 3. There may be a bug in HDFS which results in some of the data directories
> receiving corrupt data, but not all. This is the least likely use case.
> *Proposed changes to normal NN startup*
> * We should try a different FSImage if we can't load the first one we try.
> * We should examine other FSEditLogs if we can't load the first one(s) we try.
> * We should fail if we can't find EditLogs that would bring us up to what we
> believe is the latest transaction ID.
> Proposed changes to recovery mode NN startup:
> we should list out all the available storage directories and allow the
> operator to select which one he wants to use.
> Something like this:
> {code}
> Multiple storage directories found.
> 1. /foo/bar
> edits__curent__XYZ size:213421345 md5:2345345
> image size:213421345 md5:2345345
> 2. /foo/baz
> edits__curent__XYZ size:213421345 md5:2345345345
> image size:213421345 md5:2345345
> Which one would you like to use? (1/2)
> {code}
> As usual in recovery mode, we want to be flexible about error handling. In
> this case, this means that we should NOT fail if we can't find EditLogs that
> would bring us up to what we believe is the latest transaction ID.
> *Not addressed by this feature*
> This feature will not address the case where an attempt to access the
> NameNode name directory or directories hangs because of an I/O error. This
> may happen, for example, when trying to load an image from a hard-mounted NFS
> directory, when the NFS server has gone away. Just as now, the operator will
> have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3134: --- Affects Version/s: 0.23.0 Fix Version/s: 2.0.0 > harden edit log loader against malformed or malicious input > --- > > Key: HDFS-3134 > URL: https://issues.apache.org/jira/browse/HDFS-3134 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.23.0 >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Fix For: 2.0.0 > > Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, > HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch > > > Currently, the edit log loader does not handle bad or malicious input > sensibly. > We can often cause OutOfMemory exceptions, null pointer exceptions, or other > unchecked exceptions to be thrown by feeding the edit log loader bad input. > In some environments, an out of memory error can cause the JVM process to be > terminated. > It's clear that we want these exceptions to be thrown as IOException instead > of as unchecked exceptions. We also want to avoid out of memory situations. > The main task here is to put a sensible upper limit on the lengths of arrays > and strings we allocate on command. The other task is to try to avoid > creating unchecked exceptions (by dereferencing potentially-NULL pointers, > for example). Instead, we should verify ahead of time and give a more > sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253903#comment-13253903 ] Todd Lipcon commented on HDFS-2708: --- +1 pending jenkins > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2708: - Attachment: HDFS-2708.patch Here's a patch for trunk which adds a "numBlocks" field to each DN in the "LiveNodes" field of the NameNodeInfo JMX interface. > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2708: - Status: Patch Available (was: Open) > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > Attachments: HDFS-2708.patch > > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253897#comment-13253897 ] Aaron T. Myers commented on HDFS-3273: -- Hi Nicholas, could you please add some motivation for this change? Perhaps in the "Description" field of this JIRA? Thanks a lot. > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 3.0.0 > > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3279) One of the FSEditLog constructor should be moved to TestEditLog
One of the FSEditLog constructor should be moved to TestEditLog --- Key: HDFS-3279 URL: https://issues.apache.org/jira/browse/HDFS-3279 Project: Hadoop HDFS Issue Type: Improvement Components: name-node Reporter: Tsz Wo (Nicholas), SZE Assignee: Arpit Gupta Priority: Minor The FSEditLog constructor with @VisibleForTesting is used only in TestEditLog. It could be simply declared as a static method in TestEditLog. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2708) Stats for the # of blocks per DN
[ https://issues.apache.org/jira/browse/HDFS-2708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2708: - Description: It would be useful for tools to be able to retrieve the total number of blocks on each datanode. (was: It would be useful for tools to be able to retrieve the total # blocks in the file system (and also display eg via dfsadmin report, this is currently only available via FSNamesystemMetrics, so would add to ClientProtocol#getStats?) and the total number of blocks on each datanode (via DataNodeInfo).) Target Version/s: 2.0.0 (was: 1.1.0, 0.23.1) Affects Version/s: 2.0.0 Assignee: Aaron T. Myers Summary: Stats for the # of blocks per DN (was: Stats for the total # blocks and blocks per DN) Let's separate the original description of this JIRA into multiple distinct issues. In this JIRA, let's just add support for getting the number of blocks on each DN via the FSNamesystem metrics, since that interface already has support for retrieving the total number of blocks in the file system. If we really do want to add other interfaces to retrieve both of these metrics, let's do that in a separate JIRA. > Stats for the # of blocks per DN > > > Key: HDFS-2708 > URL: https://issues.apache.org/jira/browse/HDFS-2708 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node, name-node >Affects Versions: 2.0.0 >Reporter: Eli Collins >Assignee: Aaron T. Myers >Priority: Minor > > It would be useful for tools to be able to retrieve the total number of > blocks on each datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253888#comment-13253888 ] Hudson commented on HDFS-2765: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2087 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2087/]) Move CHANGES.txt message for HDFS-2765 to be under 2.0.0 instead of trunk. (Revision 1326013) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326013 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253889#comment-13253889 ] Hudson commented on HDFS-3273: -- Integrated in Hadoop-Mapreduce-trunk-Commit #2087 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2087/]) HDFS-3273. Refactor BackupImage and FSEditLog, and rename JournalListener.rollLogs(..) to startLogSegment(..). (Revision 1326016) Result = FAILURE szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326016 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalListener.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/journalservice/TestJournalService.java > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 3.0.0 > > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3042) Automatic failover support for NN HA
[ https://issues.apache.org/jira/browse/HDFS-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253883#comment-13253883 ] Todd Lipcon commented on HDFS-3042: --- Sanjay: this is being done in a branch... it's in branches/HDFS-3042 in SVN. > Automatic failover support for NN HA > > > Key: HDFS-3042 > URL: https://issues.apache.org/jira/browse/HDFS-3042 > Project: Hadoop HDFS > Issue Type: New Feature > Components: auto-failover, ha >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > HDFS-1623 was the umbrella task for implementation of NN HA capabilities. > However, it only focused on manually-triggered failover. > Given that the HDFS-1623 branch will be merged shortly, I'm opening this JIRA > to consolidate/track subtasks for automatic failover support and related > improvements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253878#comment-13253878 ] Hudson commented on HDFS-3273: -- Integrated in Hadoop-Common-trunk-Commit #2073 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2073/]) HDFS-3273. Refactor BackupImage and FSEditLog, and rename JournalListener.rollLogs(..) to startLogSegment(..). (Revision 1326016) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326016 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalListener.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/journalservice/TestJournalService.java > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 3.0.0 > > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253877#comment-13253877 ] Hudson commented on HDFS-2765: -- Integrated in Hadoop-Common-trunk-Commit #2073 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2073/]) Move CHANGES.txt message for HDFS-2765 to be under 2.0.0 instead of trunk. (Revision 1326013) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326013 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253875#comment-13253875 ] Hudson commented on HDFS-2765: -- Integrated in Hadoop-Hdfs-trunk-Commit #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2146/]) Move CHANGES.txt message for HDFS-2765 to be under 2.0.0 instead of trunk. (Revision 1326013) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326013 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253876#comment-13253876 ] Hudson commented on HDFS-3273: -- Integrated in Hadoop-Hdfs-trunk-Commit #2146 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2146/]) HDFS-3273. Refactor BackupImage and FSEditLog, and rename JournalListener.rollLogs(..) to startLogSegment(..). (Revision 1326016) Result = SUCCESS szetszwo : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1326016 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalListener.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/journalservice/JournalService.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/BackupNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/journalservice/TestJournalService.java > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 3.0.0 > > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3273: - Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Suresh, thanks for reviewing this. I have committed this. > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Fix For: 3.0.0 > > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3278) Umbrella Jira for HDFS-HA Phase 2
Umbrella Jira for HDFS-HA Phase 2 - Key: HDFS-3278 URL: https://issues.apache.org/jira/browse/HDFS-3278 Project: Hadoop HDFS Issue Type: New Feature Reporter: Sanjay Radia Assignee: Sanjay Radia HDFS-1623 gives a high level architecture and design for hot automatic failover of the NN. Branch HDFS-1623 was merged into trunk for tactical reasons even though the work for HA was not complete, Branch HDFS-1623 contained mechanisms for keeping a standby Hot (ie read from shared journal), dual block reports, fencing of DNs, Zookeeper library for leader election etc. This Umbrella jira covers the remaining work for HA and will link all the jiras for the remaining work. Unlike HDFS-1623 no single branch will be created - work will proceed in parallel branches. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers resolved HDFS-2765. -- Resolution: Fixed Fix Version/s: 2.0.0 I've back-ported this fix to branch-2. > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253869#comment-13253869 ] Suresh Srinivas commented on HDFS-3273: --- +1 for the patch. > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3042) Automatic failover support for NN HA
[ https://issues.apache.org/jira/browse/HDFS-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253867#comment-13253867 ] Sanjay Radia commented on HDFS-3042: Todd, I would have preferred to have all these done in branch as with the rest of the HA work. There is work that remains and this is best done in a branch. > Automatic failover support for NN HA > > > Key: HDFS-3042 > URL: https://issues.apache.org/jira/browse/HDFS-3042 > Project: Hadoop HDFS > Issue Type: New Feature > Components: auto-failover, ha >Reporter: Todd Lipcon >Assignee: Todd Lipcon > > HDFS-1623 was the umbrella task for implementation of NN HA capabilities. > However, it only focused on manually-triggered failover. > Given that the HDFS-1623 branch will be merged shortly, I'm opening this JIRA > to consolidate/track subtasks for automatic failover support and related > improvements. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-3274. -- Resolution: Fixed Fix Version/s: Shared journals (HDFS-3092) Hadoop Flags: Reviewed I have committed this. Thanks, Hari! > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Fix For: Shared journals (HDFS-3092) > > Attachments: hdfs-3274-1.patch, hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3049) During the normal loading NN startup process, fall back on a different EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3049:
---
Summary: During the normal loading NN startup process, fall back on a
different EditLog if we see one that is corrupt (was: During the normal
loading NN startup process, fall back on a different image or EditLog if we see
one that is corrupt)
remove references to FSImage (there is now a separate JIRA for that)
> During the normal loading NN startup process, fall back on a different
> EditLog if we see one that is corrupt
>
>
> Key: HDFS-3049
> URL: https://issues.apache.org/jira/browse/HDFS-3049
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-3049.001.patch
>
>
> During the NameNode startup process, we load an image, and then apply edit
> logs to it until we believe that we have all the latest changes.
> Unfortunately, if there is an I/O error while reading any of these files, in
> most cases, we simply abort the startup process. We should try harder to
> locate a readable edit log and/or image file.
> *There are three main use cases for this feature:*
> 1. If the operating system does not honor fsync (usually due to a
> misconfiguration), a file may end up in an inconsistent state.
> 2. In certain older releases where we did not use fallocate() or similar to
> pre-reserve blocks, a disk full condition may cause a truncated log in one
> edit directory.
> 3. There may be a bug in HDFS which results in some of the data directories
> receiving corrupt data, but not all. This is the least likely use case.
> *Proposed changes to normal NN startup*
> * We should try a different FSImage if we can't load the first one we try.
> * We should examine other FSEditLogs if we can't load the first one(s) we try.
> * We should fail if we can't find EditLogs that would bring us up to what we
> believe is the latest transaction ID.
> Proposed changes to recovery mode NN startup:
> we should list out all the available storage directories and allow the
> operator to select which one he wants to use.
> Something like this:
> {code}
> Multiple storage directories found.
> 1. /foo/bar
> edits__curent__XYZ size:213421345 md5:2345345
> image size:213421345 md5:2345345
> 2. /foo/baz
> edits__curent__XYZ size:213421345 md5:2345345345
> image size:213421345 md5:2345345
> Which one would you like to use? (1/2)
> {code}
> As usual in recovery mode, we want to be flexible about error handling. In
> this case, this means that we should NOT fail if we can't find EditLogs that
> would bring us up to what we believe is the latest transaction ID.
> *Not addressed by this feature*
> This feature will not address the case where an attempt to access the
> NameNode name directory or directories hangs because of an I/O error. This
> may happen, for example, when trying to load an image from a hard-mounted NFS
> directory, when the NFS server has gone away. Just as now, the operator will
> have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3277) fail over to loading a different FSImage if the first one we try to load is corrupt
fail over to loading a different FSImage if the first one we try to load is corrupt --- Key: HDFS-3277 URL: https://issues.apache.org/jira/browse/HDFS-3277 Project: Hadoop HDFS Issue Type: Bug Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Most users store multiple copies of the FSImage in order to prevent catastrophic data loss if a hard disk fails. However, our image loading code is currently not set up to start reading another FSImage if loading the first one does not succeed. We should add this capability. We should also be sure to remove the FSImage directory that failed from the list of FSImage directories to write to, in the way we normally do when a write (as opopsed to read) fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3049) During the normal loading NN startup process, fall back on a different image or EditLog if we see one that is corrupt
[
https://issues.apache.org/jira/browse/HDFS-3049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Colin Patrick McCabe updated HDFS-3049:
---
Attachment: HDFS-3049.001.patch
* implement edit log failover (no image stuff in here)
> During the normal loading NN startup process, fall back on a different image
> or EditLog if we see one that is corrupt
> -
>
> Key: HDFS-3049
> URL: https://issues.apache.org/jira/browse/HDFS-3049
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
>Reporter: Colin Patrick McCabe
>Assignee: Colin Patrick McCabe
>Priority: Minor
> Fix For: 0.24.0
>
> Attachments: HDFS-3049.001.patch
>
>
> During the NameNode startup process, we load an image, and then apply edit
> logs to it until we believe that we have all the latest changes.
> Unfortunately, if there is an I/O error while reading any of these files, in
> most cases, we simply abort the startup process. We should try harder to
> locate a readable edit log and/or image file.
> *There are three main use cases for this feature:*
> 1. If the operating system does not honor fsync (usually due to a
> misconfiguration), a file may end up in an inconsistent state.
> 2. In certain older releases where we did not use fallocate() or similar to
> pre-reserve blocks, a disk full condition may cause a truncated log in one
> edit directory.
> 3. There may be a bug in HDFS which results in some of the data directories
> receiving corrupt data, but not all. This is the least likely use case.
> *Proposed changes to normal NN startup*
> * We should try a different FSImage if we can't load the first one we try.
> * We should examine other FSEditLogs if we can't load the first one(s) we try.
> * We should fail if we can't find EditLogs that would bring us up to what we
> believe is the latest transaction ID.
> Proposed changes to recovery mode NN startup:
> we should list out all the available storage directories and allow the
> operator to select which one he wants to use.
> Something like this:
> {code}
> Multiple storage directories found.
> 1. /foo/bar
> edits__curent__XYZ size:213421345 md5:2345345
> image size:213421345 md5:2345345
> 2. /foo/baz
> edits__curent__XYZ size:213421345 md5:2345345345
> image size:213421345 md5:2345345
> Which one would you like to use? (1/2)
> {code}
> As usual in recovery mode, we want to be flexible about error handling. In
> this case, this means that we should NOT fail if we can't find EditLogs that
> would bring us up to what we believe is the latest transaction ID.
> *Not addressed by this feature*
> This feature will not address the case where an attempt to access the
> NameNode name directory or directories hangs because of an I/O error. This
> may happen, for example, when trying to load an image from a hard-mounted NFS
> directory, when the NFS server has gone away. Just as now, the operator will
> have to notice this problem and take steps to correct it.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers reopened HDFS-2765: -- Reopening for back-port to branch-2. > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2765) TestNameEditsConfigs is incorrectly swallowing IOE
[ https://issues.apache.org/jira/browse/HDFS-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-2765: - Target Version/s: 2.0.0 (was: 0.24.0) Affects Version/s: (was: 0.24.0) 2.0.0 Fix Version/s: (was: 3.0.0) > TestNameEditsConfigs is incorrectly swallowing IOE > -- > > Key: HDFS-2765 > URL: https://issues.apache.org/jira/browse/HDFS-2765 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Attachments: HDFS-2765.patch > > > The final portion of this test case is swallowing an IOE and in so doing > appearing to succeed, although it should not be succeeding as-written. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-3274: --- Attachment: hdfs-3274-1.patch > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274-1.patch, hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253851#comment-13253851 ] Hari Mankude commented on HDFS-3274: @Brandon done, removed the invalid comment @Nicholas done. created javadoc. Will update the patch shortly. > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253848#comment-13253848 ] Tsz Wo (Nicholas), SZE commented on HDFS-3274: -- +1 patch looks good. Please change initializeJournalStorage(..) private and the comment above to javadoc. > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3276) initializeSharedEdits should have a -nonInteractive flag
initializeSharedEdits should have a -nonInteractive flag Key: HDFS-3276 URL: https://issues.apache.org/jira/browse/HDFS-3276 Project: Hadoop HDFS Issue Type: Improvement Components: ha, name-node Affects Versions: 2.0.0 Reporter: Vinithra Varadharajan Priority: Minor Similar to format and bootstrapStandby, would be nice to have -nonInteractive as an option on initializeSharedEdits -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
[ https://issues.apache.org/jira/browse/HDFS-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aaron T. Myers updated HDFS-3275: - Description: To reproduce: # Configure a NameNode with namedirs and a shared edits dir, all of which are empty. # Run hdfs namenode -format. Namedirs and shared edits dir gets populated. # Delete the contents of the namedirs. Leave the shared edits dir as is. Check the timestamps of the shared edits dir contents. # Run format again. The namedirs as well as the shared edits dir get formatted. The shared edits dir's contents have been replaced without any prompting. was: To reproduce: 1) Configure a NameNode with namedirs and a shared edits dir, all of which are empty. 2) Run hdfs namenode -format. Namedirs and shared edits dir gets populated. 2) Delete the contents of the namedirs. Leave the shared edits dir as is. Check the timestamps of the shared edits dir contents. 3) Run format again. The namedirs as well as the shared edits dir get formatted. The shared edits dir's contents have been replaced without any prompting. Updating the numbering of the steps. (The original numbering had two step 2's.) > Format command overwrites contents of non-empty shared edits dir if name dirs > are empty without any prompting > - > > Key: HDFS-3275 > URL: https://issues.apache.org/jira/browse/HDFS-3275 > Project: Hadoop HDFS > Issue Type: Bug > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Vinithra Varadharajan > > To reproduce: > # Configure a NameNode with namedirs and a shared edits dir, all of which are > empty. > # Run hdfs namenode -format. Namedirs and shared edits dir gets populated. > # Delete the contents of the namedirs. Leave the shared edits dir as is. > Check the timestamps of the shared edits dir contents. > # Run format again. The namedirs as well as the shared edits dir get > formatted. The shared edits dir's contents have been replaced without any > prompting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3275) Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting
Format command overwrites contents of non-empty shared edits dir if name dirs are empty without any prompting - Key: HDFS-3275 URL: https://issues.apache.org/jira/browse/HDFS-3275 Project: Hadoop HDFS Issue Type: Bug Components: ha, name-node Affects Versions: 2.0.0 Reporter: Vinithra Varadharajan To reproduce: 1) Configure a NameNode with namedirs and a shared edits dir, all of which are empty. 2) Run hdfs namenode -format. Namedirs and shared edits dir gets populated. 2) Delete the contents of the namedirs. Leave the shared edits dir as is. Check the timestamps of the shared edits dir contents. 3) Run format again. The namedirs as well as the shared edits dir get formatted. The shared edits dir's contents have been replaced without any prompting. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3261) TestHASafeMode fails on HDFS-3042 branch
[ https://issues.apache.org/jira/browse/HDFS-3261?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Todd Lipcon resolved HDFS-3261. --- Resolution: Fixed Fix Version/s: Auto failover (HDFS-3042) Hadoop Flags: Reviewed Committed this yesterday, just forgot to resolve > TestHASafeMode fails on HDFS-3042 branch > > > Key: HDFS-3261 > URL: https://issues.apache.org/jira/browse/HDFS-3261 > Project: Hadoop HDFS > Issue Type: Bug > Components: test >Affects Versions: Auto failover (HDFS-3042) >Reporter: Todd Lipcon >Assignee: Todd Lipcon >Priority: Trivial > Fix For: Auto failover (HDFS-3042) > > Attachments: hdfs-3261.txt > > > TestHASafeMode started failing on the HDFS-3042 branch after the commit of > HADOOP-8247. The reason is that testEnterSafeModeInANNShouldNotThrowNPE > restarts the active node, and then tries to make an RPC to it right after > restarting. The RPC picks up a cached connection to the old (restarted) NN, > which causes an EOFException. This was just due to a test change that was > made in HADOOP-8247, not due to any change made by the actual patch. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253816#comment-13253816
]
Hudson commented on HDFS-2799:
--
Integrated in Hadoop-Mapreduce-trunk-Commit #2085 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2085/])
HDFS-2799. Trim fs.checkpoint.dir values. Contributed by Amith D K
(Revision 1325963)
Result = ABORTED
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325963
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Fix For: 2.0.0
>
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253812#comment-13253812 ] Colin Patrick McCabe commented on HDFS-3271: I guess we could make FUSE respect that configuration option as well. > src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc > -- > > Key: HDFS-3271 > URL: https://issues.apache.org/jira/browse/HDFS-3271 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253815#comment-13253815 ] Brandon Li commented on HDFS-3274: -- The patch looks good to me. Please remove the following comments in method formatStorage() sine they are irrelevant now: "For now, use the namespacedirs and edits dir to save the journal info..." > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3134: --- Attachment: HDFS-3134.005.patch new patch with the common stuff split out. Requires HADOOP-8275 > harden edit log loader against malformed or malicious input > --- > > Key: HDFS-3134 > URL: https://issues.apache.org/jira/browse/HDFS-3134 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, > HDFS-3134.003.patch, HDFS-3134.004.patch, HDFS-3134.005.patch > > > Currently, the edit log loader does not handle bad or malicious input > sensibly. > We can often cause OutOfMemory exceptions, null pointer exceptions, or other > unchecked exceptions to be thrown by feeding the edit log loader bad input. > In some environments, an out of memory error can cause the JVM process to be > terminated. > It's clear that we want these exceptions to be thrown as IOException instead > of as unchecked exceptions. We also want to avoid out of memory situations. > The main task here is to put a sensible upper limit on the lengths of arrays > and strings we allocate on command. The other task is to try to avoid > creating unchecked exceptions (by dereferencing potentially-NULL pointers, > for example). Instead, we should verify ahead of time and give a more > sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3134) harden edit log loader against malformed or malicious input
[ https://issues.apache.org/jira/browse/HDFS-3134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-3134: --- Status: Open (was: Patch Available) > harden edit log loader against malformed or malicious input > --- > > Key: HDFS-3134 > URL: https://issues.apache.org/jira/browse/HDFS-3134 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe > Attachments: HDFS-3134.001.patch, HDFS-3134.002.patch, > HDFS-3134.003.patch, HDFS-3134.004.patch > > > Currently, the edit log loader does not handle bad or malicious input > sensibly. > We can often cause OutOfMemory exceptions, null pointer exceptions, or other > unchecked exceptions to be thrown by feeding the edit log loader bad input. > In some environments, an out of memory error can cause the JVM process to be > terminated. > It's clear that we want these exceptions to be thrown as IOException instead > of as unchecked exceptions. We also want to avoid out of memory situations. > The main task here is to put a sensible upper limit on the lengths of arrays > and strings we allocate on command. The other task is to try to avoid > creating unchecked exceptions (by dereferencing potentially-NULL pointers, > for example). Instead, we should verify ahead of time and give a more > sensible error message that reflects the problem with the input. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253780#comment-13253780
]
Hudson commented on HDFS-2799:
--
Integrated in Hadoop-Hdfs-trunk-Commit #2144 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2144/])
HDFS-2799. Trim fs.checkpoint.dir values. Contributed by Amith D K
(Revision 1325963)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325963
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Fix For: 2.0.0
>
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253777#comment-13253777
]
Hudson commented on HDFS-2799:
--
Integrated in Hadoop-Common-trunk-Commit #2071 (See
[https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2071/])
HDFS-2799. Trim fs.checkpoint.dir values. Contributed by Amith D K
(Revision 1325963)
Result = SUCCESS
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325963
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSImage.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestNameEditsConfigs.java
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Fix For: 2.0.0
>
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins updated HDFS-2799:
--
Resolution: Fixed
Fix Version/s: 2.0.0
Target Version/s: (was: 0.23.1)
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed this and merged to branch-2. Thanks Amith!
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Fix For: 2.0.0
>
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253758#comment-13253758
]
Eli Collins commented on HDFS-2799:
---
+1 looks good. Two small nits: the first line of the test javadoc was indented
incorrectly and the test name had a spelling mistake ("trimm"), I'll fix those
and commit.
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eli Collins reassigned HDFS-2799:
-
Assignee: amith
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
>Assignee: amith
> Labels: noob
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-3274: --- Attachment: hdfs-3274.patch > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-3274: --- Attachment: (was: hdfs-3274.patch) > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3196) Implement JournalListener for writing journal to local disk
[ https://issues.apache.org/jira/browse/HDFS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253755#comment-13253755 ] Tsz Wo (Nicholas), SZE commented on HDFS-3196: -- Hari's patch in HDFS-3274 has editlog initialization. We should simply use it in JournalDiskWriter. > Implement JournalListener for writing journal to local disk > --- > > Key: HDFS-3196 > URL: https://issues.apache.org/jira/browse/HDFS-3196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3196_20120412.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
[ https://issues.apache.org/jira/browse/HDFS-3274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hari Mankude updated HDFS-3274: --- Attachment: hdfs-3274.patch > Create a new journal_edits_dir key to support journal nodes > > > Key: HDFS-3274 > URL: https://issues.apache.org/jira/browse/HDFS-3274 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Hari Mankude >Assignee: Hari Mankude > Attachments: hdfs-3274.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253745#comment-13253745 ] Hadoop QA commented on HDFS-3273: - +1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522608/h3273_20120413.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2273//testReport/ Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2273//console This message is automatically generated. > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2704) NameNodeResouceChecker#checkAvailableResources should check for inodes
[ https://issues.apache.org/jira/browse/HDFS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253742#comment-13253742 ] Aaron T. Myers commented on HDFS-2704: -- bq. having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing. Certainly true, though it seems unlikely that this would result in a truly corrupt edit log that the NN couldn't restart from, since running out of inodes would presumably result in no file being created at all, which isn't an invalid edit log. bq. NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant. +1 > NameNodeResouceChecker#checkAvailableResources should check for inodes > -- > > Key: HDFS-2704 > URL: https://issues.apache.org/jira/browse/HDFS-2704 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Eli Collins >Assignee: SreeHari > > NameNodeResouceChecker#checkAvailableResources currently just checks for free > space. However inodes are also a file system resource that needs to be > available (you can run out of inodes but have free space). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253730#comment-13253730 ] Hadoop QA commented on HDFS-3201: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522598/HDFS-3201.HDFS-3092.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2274//console This message is automatically generated. > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3201: - Status: Patch Available (was: Open) > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3136) Multiple SLF4J binding warning
[ https://issues.apache.org/jira/browse/HDFS-3136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3136: - Fix Version/s: 0.23.3 Merged to 0.23. > Multiple SLF4J binding warning > -- > > Key: HDFS-3136 > URL: https://issues.apache.org/jira/browse/HDFS-3136 > Project: Hadoop HDFS > Issue Type: Bug > Components: build >Affects Versions: 0.23.0 >Reporter: Jason Lowe >Assignee: Jason Lowe > Fix For: 0.23.3, 2.0.0, 3.0.0 > > Attachments: HDFS-3136.patch > > > This is the HDFS portion of HADOOP-8005. HDFS no longer depends upon slf4j, > so removing it from the assembly will eliminate the HDFS-portion of the > multiple SLF4J warnings. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3274) Create a new journal_edits_dir key to support journal nodes
Create a new journal_edits_dir key to support journal nodes Key: HDFS-3274 URL: https://issues.apache.org/jira/browse/HDFS-3274 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Hari Mankude Assignee: Hari Mankude -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3196) Implement JournalListener for writing journal to local disk
[ https://issues.apache.org/jira/browse/HDFS-3196?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253704#comment-13253704 ] Tsz Wo (Nicholas), SZE commented on HDFS-3196: -- I created HDFS-3273 for refactoring. Then, I am going to work on editlog initialization here. > Implement JournalListener for writing journal to local disk > --- > > Key: HDFS-3196 > URL: https://issues.apache.org/jira/browse/HDFS-3196 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3196_20120412.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3273: - Status: Patch Available (was: Open) > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3273) Refactor BackupImage and FSEditLog
[ https://issues.apache.org/jira/browse/HDFS-3273?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-3273: - Attachment: h3273_20120413.patch h3273_20120413.patch: - moves some code from BackupImage to FSEditLog; - renames JournalListener.rollLogs(..) to startLogSegment(..). > Refactor BackupImage and FSEditLog > -- > > Key: HDFS-3273 > URL: https://issues.apache.org/jira/browse/HDFS-3273 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Tsz Wo (Nicholas), SZE >Assignee: Tsz Wo (Nicholas), SZE > Attachments: h3273_20120413.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3273) Refactor BackupImage and FSEditLog
Refactor BackupImage and FSEditLog -- Key: HDFS-3273 URL: https://issues.apache.org/jira/browse/HDFS-3273 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Tsz Wo (Nicholas), SZE Assignee: Tsz Wo (Nicholas), SZE -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3272) Make it possible to state MIME type for a webhdfs OPEN operation's result
Make it possible to state MIME type for a webhdfs OPEN operation's result
-
Key: HDFS-3272
URL: https://issues.apache.org/jira/browse/HDFS-3272
Project: Hadoop HDFS
Issue Type: Improvement
Affects Versions: 1.0.1
Reporter: Steve Loughran
Priority: Minor
when you do a GET from the browser with webhdfs, you get the file, but it comes
over as a binary as the browser doesn't know what type it is. Having a mime
mapping table and such like would be one solution, but another is simply to add
a {{mime}} query parameter that would provide a string to be reflected back to
the caller as the Content-Type header in the HTTP response.
e.g.
{code}
http://ranier:50070/webhdfs/v1/results/Debounce/part-r-0.csv?op=open&mime=text/csv
{code}
would generate a 307 redirect to the datanode, with the
{code}
http://dn1:50075/webhdfs/v1/results/Debounce/part-r-0.csv?op=open&mime=text/csv
{code}
which would then generate the result
{code}
200 OK
Content-Type:text/csv
GATE4,eb8bd736445f415e18886ba037f84829,55000,2007-01-14,14:01:54,
GATE4,ec58edcce1049fa665446dc1fa690638,8030803000,2007-01-14,13:52:31,
...
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3201) Implement Journal reader for reading transactions from local edits log or remote Journal node
[ https://issues.apache.org/jira/browse/HDFS-3201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li updated HDFS-3201: - Attachment: HDFS-3201.HDFS-3092.patch The current plan is to use http server to transfer finalized edit log segments. This patch includes the following changes: 1. create a http server for journal node 2. have a new get image severlet to be used only by journal node 3. have a test to test file copy between journal nodes > Implement Journal reader for reading transactions from local edits log or > remote Journal node > - > > Key: HDFS-3201 > URL: https://issues.apache.org/jira/browse/HDFS-3201 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha, name-node >Reporter: Brandon Li >Assignee: Brandon Li > Attachments: HDFS-3201.HDFS-3092.patch > > -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253579#comment-13253579 ] Hadoop QA commented on HDFS-3194: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12522585/HDFS-3194.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 1 new or modified test files. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.TestSetrepDecreasing org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/2272//testReport/ Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/2272//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2272//console This message is automatically generated. > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253576#comment-13253576 ] Todd Lipcon commented on HDFS-3271: --- See HADOOP-7156. the reentrant functions are broken on some OSes so we switched to locking and non-reentrant calls. > src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc > -- > > Key: HDFS-3271 > URL: https://issues.apache.org/jira/browse/HDFS-3271 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253544#comment-13253544
]
Hadoop QA commented on HDFS-2799:
-
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12522584/HDFS-2799.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test
files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/2271//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/2271//console
This message is automatically generated.
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc -- Key: HDFS-3271 URL: https://issues.apache.org/jira/browse/HDFS-3271 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Priority: Minor Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-3271) src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc
[ https://issues.apache.org/jira/browse/HDFS-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reassigned HDFS-3271: -- Assignee: Colin Patrick McCabe > src/fuse_users.c: use re-entrant versions of getpwuid, getgid, etc > -- > > Key: HDFS-3271 > URL: https://issues.apache.org/jira/browse/HDFS-3271 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Colin Patrick McCabe >Assignee: Colin Patrick McCabe >Priority: Minor > > Use the re-entrant versions of these functions rather than using locking -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-3270) run valgrind on fuse-dfs, fix any memory leaks
run valgrind on fuse-dfs, fix any memory leaks -- Key: HDFS-3270 URL: https://issues.apache.org/jira/browse/HDFS-3270 Project: Hadoop HDFS Issue Type: Improvement Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Minor run valgrind on fuse-dfs, fix any memory leaks -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2704) NameNodeResouceChecker#checkAvailableResources should check for inodes
[ https://issues.apache.org/jira/browse/HDFS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253501#comment-13253501 ] Harsh J commented on HDFS-2704: --- NameNode also does create local image/edits files upon every checkpointing and such - so the inode monitoring is surely relevant. > NameNodeResouceChecker#checkAvailableResources should check for inodes > -- > > Key: HDFS-2704 > URL: https://issues.apache.org/jira/browse/HDFS-2704 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Eli Collins >Assignee: SreeHari > > NameNodeResouceChecker#checkAvailableResources currently just checks for free > space. However inodes are also a file system resource that needs to be > available (you can run out of inodes but have free space). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253497#comment-13253497 ] amith commented on HDFS-3194: - Hi Suja, I have done analysis on the reported defect and came up with a patch. DataBlockScanner thread will scan for the new blocks for every 5 seconds(new period start), but in the scan method of BlockPoolSliceScanner finally we are rolling the logs(irrespective of entries in logs). When a new period starts we are loading the number of entries from the logs and non scanned blocks are scanned. Since there is log roll for every 5 sec after 10 seconds the previously scanned entries are lost causing the block scan for the next period. Thus we can see the time difference is 15 sec for every consecutive scan. In the provided patch I have introduced log roll only after certain number of entries in the logs are reached(which is similar to 20.x version HDFS behaviour), care is taken not to break log interface which already exist. > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253496#comment-13253496 ] amith commented on HDFS-3194: - Hi Suja, I have done analysis on the reported defect and came up with a patch. DataBlockScanner thread will scan for the new blocks for every 5 seconds(new period start), but in the scan method of BlockPoolSliceScanner finally we are rolling the logs(irrespective of entries in logs). When a new period starts we are loading the number of entries from the logs and non scanned blocks are scanned. Since there is log roll for every 5 sec after 10 seconds the previously scanned entries are lost causing the block scan for the next period. Thus we can see the time difference is 15 sec for every consecutive scan. In the provided patch I have introduced log roll only after certain number of entries in the logs are reached(which is similar to 20.x version HDFS behaviour), care is taken not to break log interface which already exist. > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253484#comment-13253484
]
amith commented on HDFS-2799:
-
Hi Harsh/Eli
I have provided a patch can u please review the same
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] amith updated HDFS-3194: Status: Patch Available (was: Open) > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3194) Continuous block scanning at DN side
[ https://issues.apache.org/jira/browse/HDFS-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] amith updated HDFS-3194: Attachment: HDFS-3194.patch > Continuous block scanning at DN side > > > Key: HDFS-3194 > URL: https://issues.apache.org/jira/browse/HDFS-3194 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 3.0.0 >Reporter: suja s >Priority: Minor > Fix For: 3.0.0 > > Attachments: HDFS-3194.patch > > > Block scanning interval by default should be taken as 21 days(3 weeks) and > each block scanning should happen once in 21 days. > Here the block is being scanned continuosly. > 2012-04-03 10:44:47,056 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:02,064 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:17,071 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for > BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473_1003 > 2012-04-03 10:45:32,079 INFO > org.apache.hadoop.hdfs.server.datanode.BlockPoolSliceScanner: Verification > succeeded for BP-241703115-xx.xx.xx.55-1333086229434:blk_-2666054955039014473 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2799) Trim fs.checkpoint.dir values
[
https://issues.apache.org/jira/browse/HDFS-2799?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
amith updated HDFS-2799:
Attachment: HDFS-2799.patch
> Trim fs.checkpoint.dir values
> -
>
> Key: HDFS-2799
> URL: https://issues.apache.org/jira/browse/HDFS-2799
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Affects Versions: 0.23.0
>Reporter: Eli Collins
> Labels: noob
> Attachments: HDFS-2799.patch, HDFS-2799.patch
>
>
> fs.checkpoint.dir values need to be trimmed like dfs.name.dir and
> dfs.data.dir values so eg the following works. This currently results in the
> directory "HADOOP_HOME/?/home/eli/hadoop/dirs3/dfs/chkpoint1" being created.
> {noformat}
>
> fs.checkpoint.dir
>
> /home/eli/hadoop/dirs3/dfs/chkpoint1,
> /home/eli/hadoop/dirs3/dfs/chkpoint2
>
>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2704) NameNodeResouceChecker#checkAvailableResources should check for inodes
[ https://issues.apache.org/jira/browse/HDFS-2704?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253392#comment-13253392 ] Robert Joseph Evans commented on HDFS-2704: --- I personally would like to see it go into safemode, just because you never know what in the system will start to have problems when you run out of inodes, or any other resource for that matter. I would hope most standard things would fail gracefully, but you never know and having a corrupted edit log because we tried to roll them when there were no inodes available is a very bad thing. > NameNodeResouceChecker#checkAvailableResources should check for inodes > -- > > Key: HDFS-2704 > URL: https://issues.apache.org/jira/browse/HDFS-2704 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.24.0 >Reporter: Eli Collins >Assignee: SreeHari > > NameNodeResouceChecker#checkAvailableResources currently just checks for free > space. However inodes are also a file system resource that needs to be > available (you can run out of inodes but have free space). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253387#comment-13253387 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Mapreduce-trunk #1048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3255) HA DFS returns wrong token service
[
https://issues.apache.org/jira/browse/HDFS-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253388#comment-13253388
]
Hudson commented on HDFS-3255:
--
Integrated in Hadoop-Mapreduce-trunk #1048 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/])
HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp.
(Revision 1325414)
Result = SUCCESS
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
> HA DFS returns wrong token service
> --
>
> Key: HDFS-3255
> URL: https://issues.apache.org/jira/browse/HDFS-3255
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs client
>Affects Versions: 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HDFS-3255.patch
>
>
> {{fs.getCanonicalService()}} must be equal to
> {{fs.getDelegationToken(renewer).getService()}}. When HA is enabled, the DFS
> token's service is a logical uri, but {{dfs.getCanonicalService()}} is only
> returning the hostname of the logical uri.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3259) NameNode#initializeSharedEdits should populate shared edits dir with edit log segments
[ https://issues.apache.org/jira/browse/HDFS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253384#comment-13253384 ] Hudson commented on HDFS-3259: -- Integrated in Hadoop-Mapreduce-trunk #1048 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1048/]) HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir with edit log segments. Contributed by Aaron T. Myers. (Revision 1325518) Result = SUCCESS atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325518 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java > NameNode#initializeSharedEdits should populate shared edits dir with edit log > segments > -- > > Key: HDFS-3259 > URL: https://issues.apache.org/jira/browse/HDFS-3259 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3259.patch, HDFS-3259.patch, HDFS-3259.patch, > HDFS-3259.patch, HDFS-3259.patch > > > Currently initializeSharedEdits formats the shared dir so that subsequent > edit log segments will be written there. However, it would be nice to > automatically populate this dir with edit log segments with transactions > going back to the last fsimage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3255) HA DFS returns wrong token service
[
https://issues.apache.org/jira/browse/HDFS-3255?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253348#comment-13253348
]
Hudson commented on HDFS-3255:
--
Integrated in Hadoop-Hdfs-trunk #1013 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/])
HDFS-3255. HA DFS returns wrong token service. Contributed by Daryn Sharp.
(Revision 1325414)
Result = FAILURE
todd : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325414
Files :
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DistributedFileSystem.java
*
/hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestDelegationTokensWithHA.java
> HA DFS returns wrong token service
> --
>
> Key: HDFS-3255
> URL: https://issues.apache.org/jira/browse/HDFS-3255
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: ha, hdfs client
>Affects Versions: 2.0.0
>Reporter: Daryn Sharp
>Assignee: Daryn Sharp
>Priority: Critical
> Fix For: 2.0.0
>
> Attachments: HDFS-3255.patch
>
>
> {{fs.getCanonicalService()}} must be equal to
> {{fs.getDelegationToken(renewer).getService()}}. When HA is enabled, the DFS
> token's service is a logical uri, but {{dfs.getCanonicalService()}} is only
> returning the hostname of the logical uri.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3256) HDFS considers blocks under-replicated if topology script is configured with only 1 rack
[ https://issues.apache.org/jira/browse/HDFS-3256?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253347#comment-13253347 ] Hudson commented on HDFS-3256: -- Integrated in Hadoop-Hdfs-trunk #1013 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/]) HDFS-3256. HDFS considers blocks under-replicated if topology script is configured with only 1 rack. Contributed by Aaron T. Myers. (Revision 1325531) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325531 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/BlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/blockmanagement/DatanodeManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlockManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/blockmanagement/TestBlocksWithNotEnoughRacks.java > HDFS considers blocks under-replicated if topology script is configured with > only 1 rack > > > Key: HDFS-3256 > URL: https://issues.apache.org/jira/browse/HDFS-3256 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch, > HDFS-3256.patch, HDFS-3256.patch, HDFS-3256.patch > > > HDFS treats the mere presence of a topology script being configured as > evidence that there are multiple racks. If there is in fact only a single > rack, the NN will try to place the blocks on at least two racks, and thus > blocks will be considered to be under-replicated. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3259) NameNode#initializeSharedEdits should populate shared edits dir with edit log segments
[ https://issues.apache.org/jira/browse/HDFS-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253344#comment-13253344 ] Hudson commented on HDFS-3259: -- Integrated in Hadoop-Hdfs-trunk #1013 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1013/]) HDFS-3259. NameNode#initializeSharedEdits should populate shared edits dir with edit log segments. Contributed by Aaron T. Myers. (Revision 1325518) Result = FAILURE atm : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1325518 Files : * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FileJournalManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestInitializeSharedEdits.java > NameNode#initializeSharedEdits should populate shared edits dir with edit log > segments > -- > > Key: HDFS-3259 > URL: https://issues.apache.org/jira/browse/HDFS-3259 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ha, name-node >Affects Versions: 2.0.0 >Reporter: Aaron T. Myers >Assignee: Aaron T. Myers > Fix For: 2.0.0 > > Attachments: HDFS-3259.patch, HDFS-3259.patch, HDFS-3259.patch, > HDFS-3259.patch, HDFS-3259.patch > > > Currently initializeSharedEdits formats the shared dir so that subsequent > edit log segments will be written there. However, it would be nice to > automatically populate this dir with edit log segments with transactions > going back to the last fsimage. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-119) logSync() may block NameNode forever.
[
https://issues.apache.org/jira/browse/HDFS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-119:
-
Component/s: name-node
Fix Version/s: 1.1.0
> logSync() may block NameNode forever.
> -
>
> Key: HDFS-119
> URL: https://issues.apache.org/jira/browse/HDFS-119
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: name-node
>Reporter: Konstantin Shvachko
>Assignee: Suresh Srinivas
> Fix For: 0.21.0, 1.1.0
>
> Attachments: HDFS-119-branch-1.0.patch, HDFS-119-branch-1.0.patch,
> HDFS-119.patch, HDFS-119.patch
>
>
> # {{FSEditLog.logSync()}} first waits until {{isSyncRunning}} is false and
> then performs syncing to file streams by calling
> {{EditLogOutputStream.flush()}}.
> If an exception is thrown after {{isSyncRunning}} is set to {{true}} all
> threads will always wait on this condition.
> An {{IOException}} may be thrown by {{EditLogOutputStream.setReadyToFlush()}}
> or a {{RuntimeException}} may be thrown by {{EditLogOutputStream.flush()}} or
> by {{processIOError()}}.
> # The loop that calls {{eStream.flush()}} for multiple
> {{EditLogOutputStream}}-s is not synchronized, which means that another
> thread may encounter an error and modify {{editStreams}} by say calling
> {{processIOError()}}. Then the iterating process in {{logSync()}} will break
> with {{IndexOutOfBoundException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-119) logSync() may block NameNode forever.
[
https://issues.apache.org/jira/browse/HDFS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13253229#comment-13253229
]
Konstantin Shvachko commented on HDFS-119:
--
Committed to branch 1.
> logSync() may block NameNode forever.
> -
>
> Key: HDFS-119
> URL: https://issues.apache.org/jira/browse/HDFS-119
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Konstantin Shvachko
>Assignee: Suresh Srinivas
> Fix For: 0.21.0
>
> Attachments: HDFS-119-branch-1.0.patch, HDFS-119-branch-1.0.patch,
> HDFS-119.patch, HDFS-119.patch
>
>
> # {{FSEditLog.logSync()}} first waits until {{isSyncRunning}} is false and
> then performs syncing to file streams by calling
> {{EditLogOutputStream.flush()}}.
> If an exception is thrown after {{isSyncRunning}} is set to {{true}} all
> threads will always wait on this condition.
> An {{IOException}} may be thrown by {{EditLogOutputStream.setReadyToFlush()}}
> or a {{RuntimeException}} may be thrown by {{EditLogOutputStream.flush()}} or
> by {{processIOError()}}.
> # The loop that calls {{eStream.flush()}} for multiple
> {{EditLogOutputStream}}-s is not synchronized, which means that another
> thread may encounter an error and modify {{editStreams}} by say calling
> {{processIOError()}}. Then the iterating process in {{logSync()}} will break
> with {{IndexOutOfBoundException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-119) logSync() may block NameNode forever.
[
https://issues.apache.org/jira/browse/HDFS-119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Konstantin Shvachko updated HDFS-119:
-
Attachment: HDFS-119-branch-1.0.patch
Updated patch to reflect current branch state. No code changes, just line
numbers.
Plus the code style change (indentation), suggested by Brandon.
> logSync() may block NameNode forever.
> -
>
> Key: HDFS-119
> URL: https://issues.apache.org/jira/browse/HDFS-119
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Konstantin Shvachko
>Assignee: Suresh Srinivas
> Fix For: 0.21.0
>
> Attachments: HDFS-119-branch-1.0.patch, HDFS-119-branch-1.0.patch,
> HDFS-119.patch, HDFS-119.patch
>
>
> # {{FSEditLog.logSync()}} first waits until {{isSyncRunning}} is false and
> then performs syncing to file streams by calling
> {{EditLogOutputStream.flush()}}.
> If an exception is thrown after {{isSyncRunning}} is set to {{true}} all
> threads will always wait on this condition.
> An {{IOException}} may be thrown by {{EditLogOutputStream.setReadyToFlush()}}
> or a {{RuntimeException}} may be thrown by {{EditLogOutputStream.flush()}} or
> by {{processIOError()}}.
> # The loop that calls {{eStream.flush()}} for multiple
> {{EditLogOutputStream}}-s is not synchronized, which means that another
> thread may encounter an error and modify {{editStreams}} by say calling
> {{processIOError()}}. Then the iterating process in {{logSync()}} will break
> with {{IndexOutOfBoundException}}.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
