[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186095#comment-13186095 ] Uma Maheswara Rao G commented on HDFS-2768: --- Warnings are not related to this patch! > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2747) HA: entering safe mode after starting SBN can NPE
[
https://issues.apache.org/jira/browse/HDFS-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186094#comment-13186094
]
Uma Maheswara Rao G commented on HDFS-2747:
---
I feel both should work fine. In your suggestion I need not add any new methods
to get info from Editlog. It sounds good for me.
Updated the patch as per the suggestion above.
> HA: entering safe mode after starting SBN can NPE
> -
>
> Key: HDFS-2747
> URL: https://issues.apache.org/jira/browse/HDFS-2747
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2747.patch
>
>
> Entering Safemode on the primary after while it's already in safemode after
> the SBN is started results in an NPE:
> {noformat}
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode get
> Safe mode is ON
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode enter
> safemode: java.lang.NullPointerException
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2747) HA: entering safe mode after starting SBN can NPE
[
https://issues.apache.org/jira/browse/HDFS-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Uma Maheswara Rao G updated HDFS-2747:
--
Attachment: HDFS-2747.patch
> HA: entering safe mode after starting SBN can NPE
> -
>
> Key: HDFS-2747
> URL: https://issues.apache.org/jira/browse/HDFS-2747
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Uma Maheswara Rao G
> Attachments: HDFS-2747.patch
>
>
> Entering Safemode on the primary after while it's already in safemode after
> the SBN is started results in an NPE:
> {noformat}
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode get
> Safe mode is ON
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode enter
> safemode: java.lang.NullPointerException
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186087#comment-13186087 ] Hadoop QA commented on HDFS-2768: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510559/HDFS-2768.patch against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). +1 core tests. The patch passed unit tests in . +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1777//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1777//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1777//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1777//console This message is automatically generated. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186079#comment-13186079 ] Uma Maheswara Rao G commented on HDFS-2768: --- @Eli, Updated the patch. In this patch, I just moved the namenode connection close section to DFSUtil and reused every where. This patch will address your comment. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Uma Maheswara Rao G updated HDFS-2768: -- Attachment: HDFS-2768.patch > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch, HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186077#comment-13186077 ] Uma Maheswara Rao G commented on HDFS-2768: --- @Brandon, Thanks for looking this issue. Please take a look at my previous comment. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2681) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HDFS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated HDFS-2681: - Attachment: HDFS-2681.HDFS-1623.patch new patch will all changes > Add ZK client for leader election > - > > Key: HDFS-2681 > URL: https://issues.apache.org/jira/browse/HDFS-2681 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, > HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, Zookeeper based Leader > Election and Monitoring Library.pdf > > > ZKClient needs to support the following capabilities: > # Ability to create a znode for co-ordinating leader election. > # Ability to monitor and receive call backs when active znode status changes. > # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2681) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HDFS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186074#comment-13186074 ] Bikas Saha commented on HDFS-2681: -- Some more comments from chat session # Change the method name to init(). Annotate it @Before. It will be automatically called before tests. # Use @Expected for tests that expect exception Done # Add class level javadoc. This is already in the second patch. If by this you mean comments before the class declaration. # #Init need not catch IOException. Just throw it. The test will fail. I know it wont throw the exception in the test. But it has to be handled to keep the compiler happy. So I handled it locally there instead of adding "throws IOException in every method of the test" # You can reduce several lines of code by using a static byte[] DATA; done # can you add test where jointElection() is called twice and the second call is NO-OP # Many times where processResult is called back to back can be in for loop this helps me walk the scenarios better than in a loop # Why should 4 errors of connection loss result in fatalError? Because the elector has tried its best to connect to Zookeeper and failed. We can revisit this based observed failures at a later time. # testStatNodeError already covers some part of testCreateNodeResultRetryBecomeActive yes. thats because it is trying to walk through a logical scenario. so I let it be. # Instead of catching InterruptedException, you can just throw it same code cleanliness as above. I know this exception will not get thrown in the test. so want to make local changes to keep the compiler happy. >>Please use System.arraycopy() instead of byte[] clone. done. >>Split process into two different methods processZkEvent and processZnodeEvent? The function is still small enough to let it be. Will do this later when more logic might get added if we do group participation. At that point processZnodeEvent itself will need division into lock znode and parent znode. >>can you add test where jointElection() is called twice and the second call is >>NO-OP it was there in test processResult callback but got changed to enterNeutralMode when I changed that test. now I enhanced testCreateNodeResultBecomeActive() to check that there is no double master call and added another test to check that there is no double slave call for expected scenarios. now all 3 states are covered. Will upload the patch with all these changes. Thanks > Add ZK client for leader election > - > > Key: HDFS-2681 > URL: https://issues.apache.org/jira/browse/HDFS-2681 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, > HDFS-2681.HDFS-1623.patch, Zookeeper based Leader Election and Monitoring > Library.pdf > > > ZKClient needs to support the following capabilities: > # Ability to create a znode for co-ordinating leader election. > # Ability to monitor and receive call backs when active znode status changes. > # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2782) HA: Support multiple shared edits dirs
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186039#comment-13186039 ] Eli Collins commented on HDFS-2782: --- #1 Expecting both independent failures (eg the dirs are one two separate mounts with different underlying volumes may fail independently) and dependent failures (if the connection to the same server is severed for example). #2 Yes, a 2nd or 3rd shared dir is just like the first. Eg if the primary or standby marked a shared dir as bad it and the standby could switch to another. > HA: Support multiple shared edits dirs > -- > > Key: HDFS-2782 > URL: https://issues.apache.org/jira/browse/HDFS-2782 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Eli Collins >Assignee: Eli Collins > > Supporting multiple shared dirs will improve availability (eg see HDFS-2769). > You may want to use multiple shared dirs on a single filer (eg for better > fault isolation) or because you want to use multiple filers/mounts. Per > HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in > EditLogTailer and add tests. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (HDFS-2655) BlockReaderLocal#skip performs unnecessary IO
[ https://issues.apache.org/jira/browse/HDFS-2655?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Brandon Li reassigned HDFS-2655: Assignee: Brandon Li > BlockReaderLocal#skip performs unnecessary IO > - > > Key: HDFS-2655 > URL: https://issues.apache.org/jira/browse/HDFS-2655 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.1 >Reporter: Eli Collins >Assignee: Brandon Li > > Per HDFS-2654 BlockReaderLocal#skip performs the skip by reading the data so > we stay in sync with checksums. This could be implemented more efficiently in > the future to skip to the beginning of the appropriate checksum chunk and > then only read to the middle of that chunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2681) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HDFS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186026#comment-13186026 ] Suresh Srinivas commented on HDFS-2681: --- Really nice job with the tests. May be you can also create a real ZK based test. This could be done in another jira. Here are the comments: Comments: # Please use System.arraycopy() instead of byte[] clone. # Split process into two different methods processZkEvent and processZnodeEvent? # Test # Change the method name to init(). Annotate it @Before. It will be automatically called before tests. # Use @Expected for tests that expect exception # Add class level javadoc. # #Init need not catch IOException. Just throw it. The test will fail. # You can reduce several lines of code by using a static byte[] DATA; # can you add test where jointElection() is called twice and the second call is NO-OP # Many times where processResult is called back to back can be in for loop # Why should 4 errors of connection loss result in fatalError? # testStatNodeError already covers some part of testCreateNodeResultRetryBecomeActive # Instead of catching InterruptedException, you can just throw it > Add ZK client for leader election > - > > Key: HDFS-2681 > URL: https://issues.apache.org/jira/browse/HDFS-2681 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, > HDFS-2681.HDFS-1623.patch, Zookeeper based Leader Election and Monitoring > Library.pdf > > > ZKClient needs to support the following capabilities: > # Ability to create a znode for co-ordinating leader election. > # Ability to monitor and receive call backs when active znode status changes. > # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2788) HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
[ https://issues.apache.org/jira/browse/HDFS-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185957#comment-13185957 ] Hadoop QA commented on HDFS-2788: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12510531/hdfs-2788.txt against trunk revision . +1 @author. The patch does not contain any @author tags. -1 tests included. The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. -1 javadoc. The javadoc tool appears to have generated 21 warning messages. +1 javac. The applied patch does not increase the total number of javac compiler warnings. +1 eclipse:eclipse. The patch built with eclipse:eclipse. -1 findbugs. The patch appears to introduce 1 new Findbugs (version 1.3.9) warnings. -1 release audit. The applied patch generated 1 release audit warnings (more than the trunk's current 0 warnings). -1 core tests. The patch failed these unit tests: org.apache.hadoop.hdfs.web.TestFSMainOperationsWebHdfs org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract +1 contrib tests. The patch passed contrib unit tests. Test results: https://builds.apache.org/job/PreCommit-HDFS-Build/1776//testReport/ Release audit warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1776//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt Findbugs warnings: https://builds.apache.org/job/PreCommit-HDFS-Build/1776//artifact/trunk/hadoop-hdfs-project/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1776//console This message is automatically generated. > HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code > - > > Key: HDFS-2788 > URL: https://issues.apache.org/jira/browse/HDFS-2788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-2788.txt > > > HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never > used. Perhaps was renamed to > DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was > written and the old one wasn't deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2768) BackupNode stop can not close proxy connections because it is not a proxy instance.
[ https://issues.apache.org/jira/browse/HDFS-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185937#comment-13185937 ] Brandon Li commented on HDFS-2768: -- Another way to fix this is to check if the proxy is closeable in RPC.stopProxy(). If it is, just close it. This way we just need to fix the problem in one place instead of multiple ones, e.g., BackupNode, EditLogOutputStream and etc. > BackupNode stop can not close proxy connections because it is not a proxy > instance. > --- > > Key: HDFS-2768 > URL: https://issues.apache.org/jira/browse/HDFS-2768 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.24.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2768.patch > > > Observe this from BackupNode tests: > java.lang.IllegalArgumentException: not a proxy instance > at java.lang.reflect.Proxy.getInvocationHandler(Unknown Source) > at org.apache.hadoop.ipc.RPC.stopProxy(RPC.java:557) > at > org.apache.hadoop.hdfs.server.namenode.BackupNode.stop(BackupNode.java:194) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testCheckpoint(TestBackupNode.java:355) > at > org.apache.hadoop.hdfs.server.namenode.TestBackupNode.testBackupNode(TestBackupNode.java:241) > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) > at java.lang.reflect.Method.invoke(Unknown Source) > at junit.framework.TestCase.runTest(TestCase.java:168) > at junit.framework.TestCase.runBare(TestCase.java:134) > at junit.framework.TestResult$1.protect(TestResult.java:110) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2789) TestHAAdmin.testFailover is failing
[ https://issues.apache.org/jira/browse/HDFS-2789?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2789: -- Attachment: hdfs-2789.txt Patch attached. > TestHAAdmin.testFailover is failing > --- > > Key: HDFS-2789 > URL: https://issues.apache.org/jira/browse/HDFS-2789 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-2789.txt > > > Recent change broke it. Need to mock getServiceState to prevent the NPE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2789) TestHAAdmin.testFailover is failing
TestHAAdmin.testFailover is failing --- Key: HDFS-2789 URL: https://issues.apache.org/jira/browse/HDFS-2789 Project: Hadoop HDFS Issue Type: Sub-task Components: ha Affects Versions: HA branch (HDFS-1623) Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2789.txt Recent change broke it. Need to mock getServiceState to prevent the NPE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2788) HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
[ https://issues.apache.org/jira/browse/HDFS-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185882#comment-13185882 ] Kihwal Lee commented on HDFS-2788: -- Yes, that seems to be the case. > HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code > - > > Key: HDFS-2788 > URL: https://issues.apache.org/jira/browse/HDFS-2788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-2788.txt > > > HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never > used. Perhaps was renamed to > DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was > written and the old one wasn't deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-2788) HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code - Key: HDFS-2788 URL: https://issues.apache.org/jira/browse/HDFS-2788 Project: Hadoop HDFS Issue Type: Improvement Components: data-node Affects Versions: 0.23.0, 0.22.0 Reporter: Eli Collins Assignee: Eli Collins Attachments: hdfs-2788.txt HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never used. Perhaps was renamed to DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was written and the old one wasn't deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2788) HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
[ https://issues.apache.org/jira/browse/HDFS-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2788: -- Attachment: hdfs-2788.txt Patch attached. > HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code > - > > Key: HDFS-2788 > URL: https://issues.apache.org/jira/browse/HDFS-2788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.22.0, 0.23.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-2788.txt > > > HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never > used. Perhaps was renamed to > DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was > written and the old one wasn't deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2788) HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code
[ https://issues.apache.org/jira/browse/HDFS-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Eli Collins updated HDFS-2788: -- Status: Patch Available (was: Open) > HdfsServerConstants#DN_KEEPALIVE_TIMEOUT is dead code > - > > Key: HDFS-2788 > URL: https://issues.apache.org/jira/browse/HDFS-2788 > Project: Hadoop HDFS > Issue Type: Improvement > Components: data-node >Affects Versions: 0.23.0, 0.22.0 >Reporter: Eli Collins >Assignee: Eli Collins > Attachments: hdfs-2788.txt > > > HDFS-941 introduced HdfsServerConstants#DN_KEEPALIVE_TIMEOUT but its never > used. Perhaps was renamed to > DFSConfigKeys#DFS_DATANODE_SOCKET_REUSE_KEEPALIVE_DEFAULT while the patch was > written and the old one wasn't deleted. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185848#comment-13185848 ] Sanjay Radia commented on HDFS-2731: I dislike the idea of pulling the image from the primary through a protocol. (But I guess it is needed for the BNN solution.) For shared storage one could require that a shared dir be configured (NFS now and HDFS in the future by the time we get Bookeeper.). > Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Eli Collins >Assignee: Eli Collins > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2681) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HDFS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bikas Saha updated HDFS-2681: - Attachment: HDFS-2681.HDFS-1623.patch Patch that takes care of review comments. > Add ZK client for leader election > - > > Key: HDFS-2681 > URL: https://issues.apache.org/jira/browse/HDFS-2681 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, > HDFS-2681.HDFS-1623.patch, Zookeeper based Leader Election and Monitoring > Library.pdf > > > ZKClient needs to support the following capabilities: > # Ability to create a znode for co-ordinating leader election. > # Ability to monitor and receive call backs when active znode status changes. > # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2681) Add ZK client for leader election
[ https://issues.apache.org/jira/browse/HDFS-2681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185757#comment-13185757 ] Bikas Saha commented on HDFS-2681: -- >> General - How is multithreaded use for this library handled? All public methods are synchronized >>Add javadoc to the class on how to use the class. Also callback interface can >>be described in more detail on when the call back is made and perhaps some >>description of what is expected of the app. notifyError() particularly needs >>better documentation on when to expect this callback. Please document the enumerations. Done in second patch >>Constructor should check for null - at least for call back passed. Otherwise >>you will get null pointer exception. Done >>joinElection() you may want to copy the byte[] data passed or at least >>document that the data[] must not be changed by the caller. Done >>#getNewZooKeeper() seems unnecessary and can be removed. Creation of >>ZooKeeper() can be moved to createConnection() it self. This is to pass in a mock zookeeper for testing >>Make member variable that are initialized only once in the constructor final. Done in second patch >>activeData could be better name for appData. All app's can pass in data (which may go into future per app nodes). Only active app's data makes it to the lock. So I think the name is good. >>Please check if all the params are documented in methods. For example >>constructor is missing one of the params in the doc. Same is true with >>exceptions thrown. Done in second patch >>quitElection() should not check zkClient non null, as terminateConnection >>already checks it. Yeah. I forgot to remove that check after I refactored stuff into the reset() method >>getActiveData() - how about not throwing KeeperException? Also >>ActiveNotFoundException should wrap the exception caught from ZK. Its hard to differentiate exceptions inside KeeperException. There is not much the elector can do about them. The only commonly expected exception would be getting leader data when no leader exists and that has been handled as part of the elector API via a new exception. > Add ZK client for leader election > - > > Key: HDFS-2681 > URL: https://issues.apache.org/jira/browse/HDFS-2681 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Suresh Srinivas >Assignee: Bikas Saha > Fix For: HA branch (HDFS-1623) > > Attachments: HDFS-2681.HDFS-1623.patch, HDFS-2681.HDFS-1623.patch, > Zookeeper based Leader Election and Monitoring Library.pdf > > > ZKClient needs to support the following capabilities: > # Ability to create a znode for co-ordinating leader election. > # Ability to monitor and receive call backs when active znode status changes. > # Ability to get information about the active node. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2731) Autopopulate standby name dirs if they're empty
[ https://issues.apache.org/jira/browse/HDFS-2731?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185751#comment-13185751 ] Todd Lipcon commented on HDFS-2731: --- bq. I am missing this: both Image and Edits should be on shared dirs (ie no need to copy image from primary - it should be available in shared dir). Since we generalized the edit log interface, we can't store images on the shared edit storage. We only get the stream abstraction for it. We could also extend the system to require a shared *image* dir, but then we couldn't support BK-based HA. (In the future when we can store checkpoints in HDFS itself we wouldn't have that issue) > Autopopulate standby name dirs if they're empty > --- > > Key: HDFS-2731 > URL: https://issues.apache.org/jira/browse/HDFS-2731 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HA branch (HDFS-1623) >Reporter: Eli Collins >Assignee: Eli Collins > > To setup a SBN we currently format the primary then manually copy the name > dirs to the SBN. The SBN should do this automatically. Specifically, on NN > startup, if HA with a shared edits dir is configured and populated, if the > SBN has empty name dirs it should downloads the image and log from the > primary (as an optimization it could copy the logs from the shared dir). If > the other NN is still in standby then it should fail to start as it does > currently. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2747) HA: entering safe mode after starting SBN can NPE
[
https://issues.apache.org/jira/browse/HDFS-2747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185749#comment-13185749
]
Todd Lipcon commented on HDFS-2747:
---
I think the best bet is to check if the log is open for write before calling
logSyncAll. If it's not open for write, we can skip logSyncAll in
enterSafeMode, right?
> HA: entering safe mode after starting SBN can NPE
> -
>
> Key: HDFS-2747
> URL: https://issues.apache.org/jira/browse/HDFS-2747
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
>Affects Versions: HA branch (HDFS-1623)
>Reporter: Eli Collins
>Assignee: Uma Maheswara Rao G
>
> Entering Safemode on the primary after while it's already in safemode after
> the SBN is started results in an NPE:
> {noformat}
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode get
> Safe mode is ON
> hadoop-0.24.0-SNAPSHOT $ ./bin/hdfs dfsadmin -safemode enter
> safemode: java.lang.NullPointerException
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2785) Update webhdfs and httpfs for host-based token support
[ https://issues.apache.org/jira/browse/HDFS-2785?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185682#comment-13185682 ] Robert Joseph Evans commented on HDFS-2785: --- >From my first pass through the code there appear to be no changes needed. I >am going to do another more in-depth look to be sure. But this might just be >closed as "Not A Problem" > Update webhdfs and httpfs for host-based token support > -- > > Key: HDFS-2785 > URL: https://issues.apache.org/jira/browse/HDFS-2785 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: name-node, security >Affects Versions: 0.24.0, 0.23.1 >Reporter: Daryn Sharp >Assignee: Robert Joseph Evans > > Need to port 205 tokens into these filesystems. Will mainly involve ensuring > code duplicated from hftp is updated accordingly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa For more information on JIRA, see: http://www.atlassian.com/software/jira
