[jira] Commented: (HDFS-1541) Not marking datanodes dead When namenode in safemode
[ https://issues.apache.org/jira/browse/HDFS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002929#comment-13002929 ] dhruba borthakur commented on HDFS-1541: I am thinking that the namenode should not mark datanodes as dead if the namenode is in safemode, irrespective of whether it is in startup-safemode or in manual-safemode. My reasoning is as follows: A couple of times, we have had failures of a few set of racks. when this happened, we put the namenode in safemode to prevent a replication storm. When the namenode loses a large chunk of datanodes, it has to spend lots of cpu resources in processing blockreports when the partitioned datanodes start rejoining the cluster; at this time it is better if we can prevent the datanodes from timing out, or else the storm of block reports causes other datanodes to timeout resulting in a never-ending cycle. > Not marking datanodes dead When namenode in safemode > > > Key: HDFS-1541 > URL: https://issues.apache.org/jira/browse/HDFS-1541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: deadnodescheck.patch > > > In a big cluster, when namenode starts up, it takes a long time for namenode > to process block reports from all datanodes. Because heartbeats processing > get delayed, some datanodes are erroneously marked as dead, then later on > they have to register again, thus wasting time. > It would speed up starting time if the checking of dead nodes is disabled > when namenode in safemode. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1726) query method for what kind of safe mode the Namenode is in
query method for what kind of safe mode the Namenode is in
--
Key: HDFS-1726
URL: https://issues.apache.org/jira/browse/HDFS-1726
Project: Hadoop HDFS
Issue Type: Improvement
Components: name-node
Affects Versions: 0.22.0
Reporter: Matt Foley
Assignee: Matt Foley
Fix For: 0.23.0
If we could differentiate between "startup safemode" vs other safemode, it
would be easier to do startup optimizations like HDFS-1295. Looking at
FSNamesystem, this can be queried, but not with a single query, and the
semantics are not reliable under future changes. Also, the FSNamesystem code
itself, internally, uses more than one way to test for manual safe mode.
Proposal is to create a status field and query method in FSNamesystem with enum
values
{NOT_IN_SAFEMODE, SAFEMODE_STARTUP, SAFEMODE_EXTENSION, SAFEMODE_MANUAL}
If in the future we add automatic fallback to safe mode, we would add value
SAFEMODE_AUTOMATIC.
This change will make it easier to do startup optimizations, and will also
allow making the safemode management code in FSNamesystem simpler and more
consistent.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1541) Not marking datanodes dead When namenode in safemode
[ https://issues.apache.org/jira/browse/HDFS-1541?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002904#comment-13002904 ] Matt Foley commented on HDFS-1541: -- Hi Hairong, This patch uses isPopulatingReplQueues() as a proxy for being in startup mode. Is it best to start de-registering datanodes as soon as the repl queues are started, or should it wait until the system actually leaves safe mode? If the issue is that it should wait until leaving safe mode, but only when it is in "startup safemode", that relates to HDFS-1726. Thanks. > Not marking datanodes dead When namenode in safemode > > > Key: HDFS-1541 > URL: https://issues.apache.org/jira/browse/HDFS-1541 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.23.0 >Reporter: Hairong Kuang >Assignee: Hairong Kuang > Fix For: 0.23.0 > > Attachments: deadnodescheck.patch > > > In a big cluster, when namenode starts up, it takes a long time for namenode > to process block reports from all datanodes. Because heartbeats processing > get delayed, some datanodes are erroneously marked as dead, then later on > they have to register again, thus wasting time. > It would speed up starting time if the checking of dead nodes is disabled > when namenode in safemode. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1725) Cleanup FSImage construction
[ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002864#comment-13002864 ] Hadoop QA commented on HDFS-1725: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472714/HDFS-1725.diff against trunk revision 1076696. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 9 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.TestDFSRollback org.apache.hadoop.hdfs.TestDFSUpgrade org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/231//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/231//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/231//console This message is automatically generated. > Cleanup FSImage construction > > > Key: HDFS-1725 > URL: https://issues.apache.org/jira/browse/HDFS-1725 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: HDFS-1725.diff > > > FSImage construction is messy. Sometimes the storagedirectories in use are > set straight away, sometimes they are not. This makes it hard for anything > under FSImage (i.e. FSEditLog) to make assumptions about what it can use. > Therefore, this patch makes FSImage set the storage directories in use during > construction, and never allows them to change. If you want to change > storagedirectories you create a new image. > Also, all the construction code should be the same with the only difference > being the parameters passed. When not passed, these should get sensible > defaults. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-307) Namenode GUI does not show actual memory usage
[ https://issues.apache.org/jira/browse/HDFS-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002842#comment-13002842 ] John George commented on HDFS-307: -- A couple of questions came up: 1. Is the "Heap Memory used" still necessary? 2. Should the name "Heap Memory really used" be changed to something more clear like "Heap Memory Active"? > Namenode GUI does not show actual memory usage > -- > > Key: HDFS-307 > URL: https://issues.apache.org/jira/browse/HDFS-307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Liu >Assignee: John George >Priority: Minor > Attachments: a.patch > > > In the namenode GUI, the showed memory usage is not the actual memory usage. > Instead, it is the memory currently allocated to Java > (Runtime.totalMemory()). That's why we see most of the time, the memory usage > is 100%. This is a concern for operation, since this is the only page we can > monitor the memory usage of a name node. Showing 100% makes the wrong > impression that there is a memory leak in name node, and the name node needs > to be restarted. > The current value should show would be Runtime.totalMemory() - > Runtime.freeMemory(). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1725) Cleanup FSImage construction
[ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1725: - Status: Patch Available (was: Open) > Cleanup FSImage construction > > > Key: HDFS-1725 > URL: https://issues.apache.org/jira/browse/HDFS-1725 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: HDFS-1725.diff > > > FSImage construction is messy. Sometimes the storagedirectories in use are > set straight away, sometimes they are not. This makes it hard for anything > under FSImage (i.e. FSEditLog) to make assumptions about what it can use. > Therefore, this patch makes FSImage set the storage directories in use during > construction, and never allows them to change. If you want to change > storagedirectories you create a new image. > Also, all the construction code should be the same with the only difference > being the parameters passed. When not passed, these should get sensible > defaults. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1725) Cleanup FSImage construction
[ https://issues.apache.org/jira/browse/HDFS-1725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1725: - Attachment: HDFS-1725.diff > Cleanup FSImage construction > > > Key: HDFS-1725 > URL: https://issues.apache.org/jira/browse/HDFS-1725 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ivan Kelly >Assignee: Ivan Kelly > Fix For: 0.23.0 > > Attachments: HDFS-1725.diff > > > FSImage construction is messy. Sometimes the storagedirectories in use are > set straight away, sometimes they are not. This makes it hard for anything > under FSImage (i.e. FSEditLog) to make assumptions about what it can use. > Therefore, this patch makes FSImage set the storage directories in use during > construction, and never allows them to change. If you want to change > storagedirectories you create a new image. > Also, all the construction code should be the same with the only difference > being the parameters passed. When not passed, these should get sensible > defaults. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-307) Namenode GUI does not show actual memory usage
[ https://issues.apache.org/jira/browse/HDFS-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002817#comment-13002817 ] John George commented on HDFS-307: -- [exec] BUILD SUCCESSFUL [exec] Total time: 51 seconds [exec] [exec] [exec] [exec] [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] -1 tests included. The patch doesn't appear to include any new or modified tests. [exec] Please justify why no new tests are needed for this patch. [exec] Also please list what manual steps were performed to verify this patch. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] +1 system test framework. The patch passed system test framework compile. [exec] [exec] [exec] [exec] [exec] == [exec] == [exec] Finished build. [exec] == [exec] === > Namenode GUI does not show actual memory usage > -- > > Key: HDFS-307 > URL: https://issues.apache.org/jira/browse/HDFS-307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Liu >Priority: Minor > Attachments: a.patch > > > In the namenode GUI, the showed memory usage is not the actual memory usage. > Instead, it is the memory currently allocated to Java > (Runtime.totalMemory()). That's why we see most of the time, the memory usage > is 100%. This is a concern for operation, since this is the only page we can > monitor the memory usage of a name node. Showing 100% makes the wrong > impression that there is a memory leak in name node, and the name node needs > to be restarted. > The current value should show would be Runtime.totalMemory() - > Runtime.freeMemory(). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-307) Namenode GUI does not show actual memory usage
[ https://issues.apache.org/jira/browse/HDFS-307?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002818#comment-13002818 ] John George commented on HDFS-307: -- "test included" failed because no tests were included. This is a namenode GUI change and does not seem worth it to add unit tests to this. > Namenode GUI does not show actual memory usage > -- > > Key: HDFS-307 > URL: https://issues.apache.org/jira/browse/HDFS-307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Liu >Priority: Minor > Attachments: a.patch > > > In the namenode GUI, the showed memory usage is not the actual memory usage. > Instead, it is the memory currently allocated to Java > (Runtime.totalMemory()). That's why we see most of the time, the memory usage > is 100%. This is a concern for operation, since this is the only page we can > monitor the memory usage of a name node. Showing 100% makes the wrong > impression that there is a memory leak in name node, and the name node needs > to be restarted. > The current value should show would be Runtime.totalMemory() - > Runtime.freeMemory(). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-307) Namenode GUI does not show actual memory usage
[ https://issues.apache.org/jira/browse/HDFS-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-307: - Assignee: John George Status: Patch Available (was: Open) > Namenode GUI does not show actual memory usage > -- > > Key: HDFS-307 > URL: https://issues.apache.org/jira/browse/HDFS-307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Liu >Assignee: John George >Priority: Minor > Attachments: a.patch > > > In the namenode GUI, the showed memory usage is not the actual memory usage. > Instead, it is the memory currently allocated to Java > (Runtime.totalMemory()). That's why we see most of the time, the memory usage > is 100%. This is a concern for operation, since this is the only page we can > monitor the memory usage of a name node. Showing 100% makes the wrong > impression that there is a memory leak in name node, and the name node needs > to be restarted. > The current value should show would be Runtime.totalMemory() - > Runtime.freeMemory(). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-307) Namenode GUI does not show actual memory usage
[ https://issues.apache.org/jira/browse/HDFS-307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] John George updated HDFS-307: - Attachment: a.patch > Namenode GUI does not show actual memory usage > -- > > Key: HDFS-307 > URL: https://issues.apache.org/jira/browse/HDFS-307 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Qi Liu >Priority: Minor > Attachments: a.patch > > > In the namenode GUI, the showed memory usage is not the actual memory usage. > Instead, it is the memory currently allocated to Java > (Runtime.totalMemory()). That's why we see most of the time, the memory usage > is 100%. This is a concern for operation, since this is the only page we can > monitor the memory usage of a name node. Showing 100% makes the wrong > impression that there is a memory leak in name node, and the name node needs > to be restarted. > The current value should show would be Runtime.totalMemory() - > Runtime.freeMemory(). -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1720) Federation: FSVolumeSet volumes is not synchronized correctly
[ https://issues.apache.org/jira/browse/HDFS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002768#comment-13002768 ] Tsz Wo (Nicholas), SZE commented on HDFS-1720: -- +1 HDFS-1720.1.patch looks good. > Federation: FSVolumeSet volumes is not synchronized correctly > -- > > Key: HDFS-1720 > URL: https://issues.apache.org/jira/browse/HDFS-1720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1720.1.patch, HDFS-1720.patch > > > Currently FSVolumeSet#volumes is package private and is exposed to outside > classes: > # Only some methods (such as FSVolumeSet#checkDirs()) are synchronized on > FSVolumeSet.this. This method changes the > content of the array (sets volumes with errors to null). > # Some access to volumes are synchronized by FSDataset.this. Some access are > not synchronized at all. > I propose making FSVolumeSet#unmodifiable list. This prevents accidental > mutation from outside the class. The volumes > also are created anew when modifications are made. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1719) Federation: TestDFSRemove fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002747#comment-13002747 ] Boris Shkolnik commented on HDFS-1719: -- +1 > Federation: TestDFSRemove fails intermittently > --- > > Key: HDFS-1719 > URL: https://issues.apache.org/jira/browse/HDFS-1719 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1719.1.patch, HDFS-1719.patch > > > This failure is due to Datanode#datanodeId being seen as null by another > thread. Datanode#datanodeId access is not > synchronized. It has many fields, datanode name, info port, ipc port, > storageID etc. These fields are some times set > and some times read without being synchronized. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1719) Federation: TestDFSRemove fails intermittently
[ https://issues.apache.org/jira/browse/HDFS-1719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1719: -- Attachment: HDFS-1719.1.patch New patch after merging conflicts. > Federation: TestDFSRemove fails intermittently > --- > > Key: HDFS-1719 > URL: https://issues.apache.org/jira/browse/HDFS-1719 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1719.1.patch, HDFS-1719.patch > > > This failure is due to Datanode#datanodeId being seen as null by another > thread. Datanode#datanodeId access is not > synchronized. It has many fields, datanode name, info port, ipc port, > storageID etc. These fields are some times set > and some times read without being synchronized. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-121) NullPointerException in INode prevent Namenode from starting
[ https://issues.apache.org/jira/browse/HDFS-121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE resolved HDFS-121. - Resolution: Not A Problem Closing. Please feel free to reopen if this is still a problem. > NullPointerException in INode prevent Namenode from starting > > > Key: HDFS-121 > URL: https://issues.apache.org/jira/browse/HDFS-121 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node > Environment: CentOS 5, Sun JDK 1.5.0_15 >Reporter: Xavier Stevens > > After a headnode went down due to a kernel panic, it was restarted. When > trying to restart the Hadoop process we encountered the following > NullPointerException. It seems this should be handled more gracefully > allowing the Name Node to come up and function while either deleting or > ignoring the problematic INodes. > 2008-07-09 14:30:11,458 INFO org.apache.hadoop.fs.FSNamesystem: > isPermissionEnabled=true > 2008-07-09 14:30:12,713 ERROR org.apache.hadoop.dfs.NameNode: > java.lang.NullPointerException > at > org.apache.hadoop.dfs.INodeDirectory.getExistingPathINodes(INode.java:408) > at org.apache.hadoop.dfs.INodeDirectory.getNode(INode.java:357) > at org.apache.hadoop.dfs.INodeDirectory.getNode(INode.java:365) > at > org.apache.hadoop.dfs.FSDirectory.unprotectedDelete(FSDirectory.java:458) > at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:537) > at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:756) > at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:639) > at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:222) > at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79) > at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:254) > at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:235) > at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131) > at org.apache.hadoop.dfs.NameNode.(NameNode.java:176) > at org.apache.hadoop.dfs.NameNode.(NameNode.java:162) > at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:846) > at org.apache.hadoop.dfs.NameNode.main(NameNode.java:855) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-121) NullPointerException in INode prevent Namenode from starting
[ https://issues.apache.org/jira/browse/HDFS-121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo (Nicholas), SZE updated HDFS-121: Component/s: name-node The affects version was 0.16.4. > NullPointerException in INode prevent Namenode from starting > > > Key: HDFS-121 > URL: https://issues.apache.org/jira/browse/HDFS-121 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node > Environment: CentOS 5, Sun JDK 1.5.0_15 >Reporter: Xavier Stevens > > After a headnode went down due to a kernel panic, it was restarted. When > trying to restart the Hadoop process we encountered the following > NullPointerException. It seems this should be handled more gracefully > allowing the Name Node to come up and function while either deleting or > ignoring the problematic INodes. > 2008-07-09 14:30:11,458 INFO org.apache.hadoop.fs.FSNamesystem: > isPermissionEnabled=true > 2008-07-09 14:30:12,713 ERROR org.apache.hadoop.dfs.NameNode: > java.lang.NullPointerException > at > org.apache.hadoop.dfs.INodeDirectory.getExistingPathINodes(INode.java:408) > at org.apache.hadoop.dfs.INodeDirectory.getNode(INode.java:357) > at org.apache.hadoop.dfs.INodeDirectory.getNode(INode.java:365) > at > org.apache.hadoop.dfs.FSDirectory.unprotectedDelete(FSDirectory.java:458) > at org.apache.hadoop.dfs.FSEditLog.loadFSEdits(FSEditLog.java:537) > at org.apache.hadoop.dfs.FSImage.loadFSEdits(FSImage.java:756) > at org.apache.hadoop.dfs.FSImage.loadFSImage(FSImage.java:639) > at org.apache.hadoop.dfs.FSImage.recoverTransitionRead(FSImage.java:222) > at org.apache.hadoop.dfs.FSDirectory.loadFSImage(FSDirectory.java:79) > at org.apache.hadoop.dfs.FSNamesystem.initialize(FSNamesystem.java:254) > at org.apache.hadoop.dfs.FSNamesystem.(FSNamesystem.java:235) > at org.apache.hadoop.dfs.NameNode.initialize(NameNode.java:131) > at org.apache.hadoop.dfs.NameNode.(NameNode.java:176) > at org.apache.hadoop.dfs.NameNode.(NameNode.java:162) > at org.apache.hadoop.dfs.NameNode.createNameNode(NameNode.java:846) > at org.apache.hadoop.dfs.NameNode.main(NameNode.java:855) -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002693#comment-13002693
]
Hadoop QA commented on HDFS-1611:
-
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12472668/HDFS-1611.patch
against trunk revision 1076696.
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified
tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac
compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9)
warnings.
+1 release audit. The applied patch does not increase the total number of
release audit warnings.
-1 core tests. The patch failed these core unit tests:
org.apache.hadoop.hdfs.TestFileConcurrentReader
-1 contrib tests. The patch failed contrib unit tests.
+1 system test framework. The patch passed system test framework compile.
Test results:
https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/230//testReport/
Findbugs warnings:
https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/230//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output:
https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/230//console
This message is automatically generated.
> Some logical issues need to address.
>
>
> Key: HDFS-1611
> URL: https://issues.apache.org/jira/browse/HDFS-1611
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, name-node
>Affects Versions: 0.20.1, 0.20.2
>Reporter: Uma Maheswara Rao G
>Priority: Minor
> Attachments: HDFS-1611.patch
>
>
> Title: Some code level logical issues.
> Description:
> 1. DFSClient:
> Consider the below case, if we enable only info, then below log will never
> be logged.
> if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
> ClientDatanodeProtocol.LOG.info("ClientDatanodeProtocol addr=" + addr);
> }
> 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
>
> catch (NotCompliantMBeanException e) {
> e.printStackTrace();
> }
> We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1609) rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
[ https://issues.apache.org/jira/browse/HDFS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002673#comment-13002673 ] Hadoop QA commented on HDFS-1609: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12472666/HDFS-1609-test.patch against trunk revision 1076696. +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 7 new or modified tests. +1 javadoc. The javadoc tool did not generate any warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. +1 release audit. The applied patch does not increase the total number of release audit warnings. -1 core tests. The patch failed these core unit tests: org.apache.hadoop.hdfs.server.datanode.TestBlockReport org.apache.hadoop.hdfs.TestDFSShell org.apache.hadoop.hdfs.TestFileConcurrentReader -1 contrib tests. The patch failed contrib unit tests. +1 system test framework. The patch passed system test framework compile. Test results: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/229//testReport/ Findbugs warnings: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/229//artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html Console output: https://hudson.apache.org/hudson/job/PreCommit-HDFS-Build/229//console This message is automatically generated. > rmr command is not displaying any error message when a path contains wildcard > characters and does not exist. > > > Key: HDFS-1609 > URL: https://issues.apache.org/jira/browse/HDFS-1609 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1, 0.20.2 >Reporter: Uma Maheswara Rao G >Priority: Minor > Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, > HDFS-1609.patch > > > When we give invalid directory path then it will show error message on the > console. But if we provide the wildcard expression in invalid directory path > then it will not show any error message even there is no pattern match for > that path. > linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test > rmr: cannot remove /test/test: No such file or directory. > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* * > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #* -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1609) rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
[ https://issues.apache.org/jira/browse/HDFS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002664#comment-13002664 ] Koji Noguchi commented on HDFS-1609: I see. tcsh and bash behaves differently for multiple globbing. > rmr command is not displaying any error message when a path contains wildcard > characters and does not exist. > > > Key: HDFS-1609 > URL: https://issues.apache.org/jira/browse/HDFS-1609 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1, 0.20.2 >Reporter: Uma Maheswara Rao G >Priority: Minor > Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, > HDFS-1609.patch > > > When we give invalid directory path then it will show error message on the > console. But if we provide the wildcard expression in invalid directory path > then it will not show any error message even there is no pattern match for > that path. > linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test > rmr: cannot remove /test/test: No such file or directory. > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* * > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #* -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-1611:
--
Status: Patch Available (was: Open)
> Some logical issues need to address.
>
>
> Key: HDFS-1611
> URL: https://issues.apache.org/jira/browse/HDFS-1611
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, name-node
>Affects Versions: 0.20.2, 0.20.1
>Reporter: Uma Maheswara Rao G
>Priority: Minor
> Attachments: HDFS-1611.patch
>
>
> Title: Some code level logical issues.
> Description:
> 1. DFSClient:
> Consider the below case, if we enable only info, then below log will never
> be logged.
> if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
> ClientDatanodeProtocol.LOG.info("ClientDatanodeProtocol addr=" + addr);
> }
> 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
>
> catch (NotCompliantMBeanException e) {
> e.printStackTrace();
> }
> We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002653#comment-13002653
]
Uma Maheswara Rao G commented on HDFS-1611:
---
Verification Result:
For ths changes tests not required.
[exec] +1 overall.
[exec]
[exec] +1 @author. The patch does not contain any @author tags.
[exec]
[exec] -1 tests included. The patch doesn't appear to include any new
or modified tests.
[exec] Please justify why no new tests are needed
for this patch.
[exec] Also please list what manual steps were
performed to verify this patch.
[exec]
[exec] +1 javadoc. The javadoc tool did not generate any warning
messages.
[exec]
[exec] +1 javac. The applied patch does not increase the total number
of javac compiler warnings.
[exec]
[exec] +1 findbugs. The patch does not introduce any new Findbugs
(version 1.3.9) warnings.
[exec]
[exec] +1 release audit. The applied patch does not increase the
total number of release audit warnings.
[exec]
[exec] +1 system test framework. The patch passed system test
framework compile.
Tests are not required for this changes.That is the reason for -1.
> Some logical issues need to address.
>
>
> Key: HDFS-1611
> URL: https://issues.apache.org/jira/browse/HDFS-1611
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, name-node
>Affects Versions: 0.20.1, 0.20.2
>Reporter: Uma Maheswara Rao G
>Priority: Minor
> Attachments: HDFS-1611.patch
>
>
> Title: Some code level logical issues.
> Description:
> 1. DFSClient:
> Consider the below case, if we enable only info, then below log will never
> be logged.
> if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
> ClientDatanodeProtocol.LOG.info("ClientDatanodeProtocol addr=" + addr);
> }
> 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
>
> catch (NotCompliantMBeanException e) {
> e.printStackTrace();
> }
> We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1609) rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
[ https://issues.apache.org/jira/browse/HDFS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002629#comment-13002629 ] Uma Maheswara Rao G commented on HDFS-1609: --- Fix for this defect is in FsShell.java but the file present in COMMON project. Patch (HDFS-1609-src.patch) need to be updated in COMMON. > rmr command is not displaying any error message when a path contains wildcard > characters and does not exist. > > > Key: HDFS-1609 > URL: https://issues.apache.org/jira/browse/HDFS-1609 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1, 0.20.2 >Reporter: Uma Maheswara Rao G >Priority: Minor > Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, > HDFS-1609.patch > > > When we give invalid directory path then it will show error message on the > console. But if we provide the wildcard expression in invalid directory path > then it will not show any error message even there is no pattern match for > that path. > linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test > rmr: cannot remove /test/test: No such file or directory. > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* * > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #* -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1609) rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
[ https://issues.apache.org/jira/browse/HDFS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-1609: -- Status: Patch Available (was: Open) > rmr command is not displaying any error message when a path contains wildcard > characters and does not exist. > > > Key: HDFS-1609 > URL: https://issues.apache.org/jira/browse/HDFS-1609 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.2, 0.20.1 >Reporter: Uma Maheswara Rao G >Priority: Minor > Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, > HDFS-1609.patch > > > When we give invalid directory path then it will show error message on the > console. But if we provide the wildcard expression in invalid directory path > then it will not show any error message even there is no pattern match for > that path. > linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test > rmr: cannot remove /test/test: No such file or directory. > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* * > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #* -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1611) Some logical issues need to address.
[
https://issues.apache.org/jira/browse/HDFS-1611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-1611:
--
Attachment: HDFS-1611.patch
> Some logical issues need to address.
>
>
> Key: HDFS-1611
> URL: https://issues.apache.org/jira/browse/HDFS-1611
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs client, name-node
>Affects Versions: 0.20.1, 0.20.2
>Reporter: Uma Maheswara Rao G
>Priority: Minor
> Attachments: HDFS-1611.patch
>
>
> Title: Some code level logical issues.
> Description:
> 1. DFSClient:
> Consider the below case, if we enable only info, then below log will never
> be logged.
> if (ClientDatanodeProtocol.LOG.isDebugEnabled()) {
> ClientDatanodeProtocol.LOG.info("ClientDatanodeProtocol addr=" + addr);
> }
> 2.org.apache.hadoop.hdfs.server.namenode.FSNamesystem.registerMBean()
>
> catch (NotCompliantMBeanException e) {
> e.printStackTrace();
> }
> We can avoid using stackTace(). Better to add log message.
--
This message is automatically generated by JIRA.
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1609) rmr command is not displaying any error message when a path contains wildcard characters and does not exist.
[ https://issues.apache.org/jira/browse/HDFS-1609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-1609: -- Attachment: HDFS-1609-test.patch HDFS-1609-src.patch > rmr command is not displaying any error message when a path contains wildcard > characters and does not exist. > > > Key: HDFS-1609 > URL: https://issues.apache.org/jira/browse/HDFS-1609 > Project: Hadoop HDFS > Issue Type: Bug > Components: hdfs client >Affects Versions: 0.20.1, 0.20.2 >Reporter: Uma Maheswara Rao G >Priority: Minor > Attachments: HDFS-1609-src.patch, HDFS-1609-test.patch, > HDFS-1609.patch > > > When we give invalid directory path then it will show error message on the > console. But if we provide the wildcard expression in invalid directory path > then it will not show any error message even there is no pattern match for > that path. > linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test/test > rmr: cannot remove /test/test: No such file or directory. > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin # ./hdfs dfs -rmr /test* * > *linux-9j5v:/home/hadoop-hdfs-0.22.0-SNAPSHOT/bin #* -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Created: (HDFS-1725) Cleanup FSImage construction
Cleanup FSImage construction Key: HDFS-1725 URL: https://issues.apache.org/jira/browse/HDFS-1725 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Ivan Kelly Assignee: Ivan Kelly Fix For: 0.23.0 FSImage construction is messy. Sometimes the storagedirectories in use are set straight away, sometimes they are not. This makes it hard for anything under FSImage (i.e. FSEditLog) to make assumptions about what it can use. Therefore, this patch makes FSImage set the storage directories in use during construction, and never allows them to change. If you want to change storagedirectories you create a new image. Also, all the construction code should be the same with the only difference being the parameters passed. When not passed, these should get sensible defaults. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1489) breaking the dependency between FSEditLog and FSImage
[ https://issues.apache.org/jira/browse/HDFS-1489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly resolved HDFS-1489. -- Resolution: Incomplete Assignee: Ivan Kelly This is now being handled by HDFS-1580 in a less monolithic fashion. > breaking the dependency between FSEditLog and FSImage > - > > Key: HDFS-1489 > URL: https://issues.apache.org/jira/browse/HDFS-1489 > Project: Hadoop HDFS > Issue Type: Improvement > Components: name-node >Affects Versions: 0.21.0 >Reporter: Diego Marron >Assignee: Ivan Kelly > Attachments: HDFS-1489.diff, HDFS-1489.diff, HDFS-1489.pdf, > NNStorage.diff, NNStorage.diff, NNStorage.diff, hdfs_1489.pdf > > > This is a refactor patch which its main concerns are: > - breaking the dependency between FSEditLog and FSImage > - Splitting the abstracting the error handling and directory management, > - Decoupling Storage from FSImage. > In order to accomplish the above goal, we will need to introduce new classes: > - NNStorage: Will care about the storage. It extends Storage class, and will > contain the StorageDirectories. > - NNUtils: Some utility static methods on FSImage and FSEditLog will be > moved here. > - PersistenceManager: FSNameSystem will now be responsible for managing the > FSImage & FSEditLog objects. There will be some logic that will have to moved > out of FSImage to facilite this. For this we propose a PersistanceManager? > object as follows. > For more deep details, see the design document uploaded. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1580) Add interface for generic Write Ahead Logging mechanisms
[ https://issues.apache.org/jira/browse/HDFS-1580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ivan Kelly updated HDFS-1580: - Issue Type: Improvement (was: Sub-task) Parent: (was: HDFS-1489) > Add interface for generic Write Ahead Logging mechanisms > > > Key: HDFS-1580 > URL: https://issues.apache.org/jira/browse/HDFS-1580 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Ivan Kelly > Attachments: HDFS-1580+1521.diff, HDFS-1580.diff, > generic_wal_iface.pdf, generic_wal_iface.pdf, generic_wal_iface.txt > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1720) Federation: FSVolumeSet volumes is not synchronized correctly
[ https://issues.apache.org/jira/browse/HDFS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002512#comment-13002512 ] Suresh Srinivas commented on HDFS-1720: --- testpatch result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.8) warnings. [exec] [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] [exec] -1 system test framework. The patch failed system test framework compile. The -1 in system test framework is not related to this patch. It is not expected after merging trunk to federation branch. > Federation: FSVolumeSet volumes is not synchronized correctly > -- > > Key: HDFS-1720 > URL: https://issues.apache.org/jira/browse/HDFS-1720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1720.1.patch, HDFS-1720.patch > > > Currently FSVolumeSet#volumes is package private and is exposed to outside > classes: > # Only some methods (such as FSVolumeSet#checkDirs()) are synchronized on > FSVolumeSet.this. This method changes the > content of the array (sets volumes with errors to null). > # Some access to volumes are synchronized by FSDataset.this. Some access are > not synchronized at all. > I propose making FSVolumeSet#unmodifiable list. This prevents accidental > mutation from outside the class. The volumes > also are created anew when modifications are made. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Updated: (HDFS-1720) Federation: FSVolumeSet volumes is not synchronized correctly
[ https://issues.apache.org/jira/browse/HDFS-1720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas updated HDFS-1720: -- Attachment: HDFS-1720.1.patch Updated patch after fixing conflicts. > Federation: FSVolumeSet volumes is not synchronized correctly > -- > > Key: HDFS-1720 > URL: https://issues.apache.org/jira/browse/HDFS-1720 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: data-node >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1720.1.patch, HDFS-1720.patch > > > Currently FSVolumeSet#volumes is package private and is exposed to outside > classes: > # Only some methods (such as FSVolumeSet#checkDirs()) are synchronized on > FSVolumeSet.this. This method changes the > content of the array (sets volumes with errors to null). > # Some access to volumes are synchronized by FSDataset.this. Some access are > not synchronized at all. > I propose making FSVolumeSet#unmodifiable list. This prevents accidental > mutation from outside the class. The volumes > also are created anew when modifications are made. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1312) Re-balance disks within a Datanode
[ https://issues.apache.org/jira/browse/HDFS-1312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002509#comment-13002509 ] Allen Wittenauer commented on HDFS-1312: >For the cluster monitoring issue, we could not expect an integrated monitoring > substitute the external monitoring system. Thus, we need to make clear > what information gathering requirements should be considered inside hdfs. There is nothing preventing us from building a jsp that runs on the datanode that shows the file systems in use and the relevant stats for those fs's. >And the "lock" I mentions is to stop write new blocks into the volume, > which is the simplest way to migrate blocks. This is going to be a big performance hit. I think the focus should be on already written/closed blocks and just ignore new blocks. > Re-balance disks within a Datanode > -- > > Key: HDFS-1312 > URL: https://issues.apache.org/jira/browse/HDFS-1312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: data-node >Reporter: Travis Crawford > > Filing this issue in response to ``full disk woes`` on hdfs-user. > Datanodes fill their storage directories unevenly, leading to situations > where certain disks are full while others are significantly less used. Users > at many different sites have experienced this issue, and HDFS administrators > are taking steps like: > - Manually rebalancing blocks in storage directories > - Decomissioning nodes & later readding them > There's a tradeoff between making use of all available spindles, and filling > disks at the sameish rate. Possible solutions include: > - Weighting less-used disks heavier when placing new blocks on the datanode. > In write-heavy environments this will still make use of all spindles, > equalizing disk use over time. > - Rebalancing blocks locally. This would help equalize disk use as disks are > added/replaced in older cluster nodes. > Datanodes should actively manage their local disk so operator intervention is > not needed. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1718) HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail
[ https://issues.apache.org/jira/browse/HDFS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-1718. --- Resolution: Fixed Hadoop Flags: [Reviewed] I committed the patch. > HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail > --- > > Key: HDFS-1718 > URL: https://issues.apache.org/jira/browse/HDFS-1718 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1718.patch > > > MiniDFSCluster#shouldWait() method waits for all the datanodes to come up and > register with the namenode. > Due to threading issues some of the tests fail for two reasons: > # Datanode#isDatanodeUp() fails even if all the BPOfferService threads have > exited. This is due to Thread.isAlive() > returning true, even though the thread has exited. Adding a check to > BPOfferService#shouldService run as an addition, > fixes this issues. > # shouldWait(), where isBPServiceAlive() is called, does not work when a > BPOfferService thread fails before the > datanode has discovered the BPID, from handshake with namenode. This can be > fixed by checking the thread state using > InetSocketAddress to determine the BPOfferService, instead of BPID. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1718) HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail
[ https://issues.apache.org/jira/browse/HDFS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002504#comment-13002504 ] Suresh Srinivas commented on HDFS-1718: --- test-patch results: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 12 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.8) warnings. [exec] [exec] -1 release audit. The applied patch generated 100 release audit warnings (more than the trunk's current 98 warnings). [exec] [exec] -1 system test framework. The patch failed system test framework compile. The -1 in release audit and system test framework is not related to this patch. It is not expected after merging trunk to federation branch. > HDFS Federation: MiniDFSCluster#waitActive() bug causes some tests to fail > --- > > Key: HDFS-1718 > URL: https://issues.apache.org/jira/browse/HDFS-1718 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Affects Versions: Federation Branch >Reporter: Suresh Srinivas >Assignee: Suresh Srinivas > Fix For: Federation Branch > > Attachments: HDFS-1718.patch > > > MiniDFSCluster#shouldWait() method waits for all the datanodes to come up and > register with the namenode. > Due to threading issues some of the tests fail for two reasons: > # Datanode#isDatanodeUp() fails even if all the BPOfferService threads have > exited. This is due to Thread.isAlive() > returning true, even though the thread has exited. Adding a check to > BPOfferService#shouldService run as an addition, > fixes this issues. > # shouldWait(), where isBPServiceAlive() is called, does not work when a > BPOfferService thread fails before the > datanode has discovered the BPID, from handshake with namenode. This can be > fixed by checking the thread state using > InetSocketAddress to determine the BPOfferService, instead of BPID. -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Resolved: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
[ https://issues.apache.org/jira/browse/HDFS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-1722. --- Resolution: Fixed Hadoop Flags: [Reviewed] I committed the patch. Thank you Boris. > HDFS Federation: Add flag to MiniDFSCluster to differentiate between two > different modes-Federation and not. > - > > Key: HDFS-1722 > URL: https://issues.apache.org/jira/browse/HDFS-1722 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1722.patch, HDFS-1722.patch > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-1722) HDFS Federation: Add flag to MiniDFSCluster to differentiate between two different modes-Federation and not.
[ https://issues.apache.org/jira/browse/HDFS-1722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13002499#comment-13002499 ] Suresh Srinivas commented on HDFS-1722: --- testpatch result: [exec] -1 overall. [exec] [exec] +1 @author. The patch does not contain any @author tags. [exec] [exec] +1 tests included. The patch appears to include 6 new or modified tests. [exec] [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.8) warnings. [exec] [exec] -1 release audit. The applied patch generated 100 release audit warnings (more than the trunk's current 98 warnings). [exec] [exec] -1 system test framework. The patch failed system test framework compile. The -1 in release audit and system test framework is not related to this patch. It is not expected after merging trunk to federation branch. > HDFS Federation: Add flag to MiniDFSCluster to differentiate between two > different modes-Federation and not. > - > > Key: HDFS-1722 > URL: https://issues.apache.org/jira/browse/HDFS-1722 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node >Reporter: Boris Shkolnik >Assignee: Boris Shkolnik > Attachments: HDFS-1722.patch, HDFS-1722.patch > > -- This message is automatically generated by JIRA. - For more information on JIRA, see: http://www.atlassian.com/software/jira
