[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105893#comment-13105893 ] Uma Maheswara Rao G commented on HDFS-2271: --- Hi Arun, Thanks for taking a look! This Code & design has been refactored in trunk. ( as part of HDFS-1073). Most of the cases handled with HDFS-1073. I am checking the trunk code considering this scenarios, if i find such cases i will file the JIRAs. Thanks Uma > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2271.1.patch, HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105863#comment-13105863 ] Arun C Murthy commented on HDFS-2271: - Is this not required for 0.23/0.24 ? > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2271.1.patch, HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13105258#comment-13105258 ] Hudson commented on HDFS-2271: -- Integrated in Hadoop-Hdfs-22-branch #84 (See [https://builds.apache.org/job/Hadoop-Hdfs-22-branch/84/]) HDFS-2271. startJournalSpool should invoke processIOError. Contributed by Uma Maheswara Rao G. shv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1170925 Files : * /hadoop/common/branches/branch-0.22/hdfs/CHANGES.txt * /hadoop/common/branches/branch-0.22/hdfs/src/java/org/apache/hadoop/hdfs/server/namenode/BackupStorage.java * /hadoop/common/branches/branch-0.22/hdfs/src/test/hdfs/org/apache/hadoop/hdfs/server/namenode/TestBackupNode.java > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Fix For: 0.22.0 > > Attachments: HDFS-2271.1.patch, HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13104132#comment-13104132 ] Hadoop QA commented on HDFS-2271: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494356/HDFS-2271.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1238//console This message is automatically generated. > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.1.patch, HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103924#comment-13103924 ] Konstantin Shvachko commented on HDFS-2271: --- Good catch. One nit. In TestBackupNode.testStartJournalSpoolHandlesIOExceptions() please use BASE_DIR rather than writing to /tmp. That way we guarantee that clean target removes everything created on the previous test run. > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103807#comment-13103807 ] Hadoop QA commented on HDFS-2271: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12494275/HDFS-2271.1.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1237//console This message is automatically generated. > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103795#comment-13103795 ] Uma Maheswara Rao G commented on HDFS-2271: --- Test patch results: [exec] +1 @author. The patch does not contain any @author tags. [exec] +1 tests included. The patch appears to include 3 new or modified tests. [exec] +1 javadoc. The javadoc tool did not generate any warning messages. [exec] +1 javac. The applied patch does not increase the total number of javac compiler warnings. [exec] +1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings. [exec] +1 release audit. The applied patch does not increase the total number of release audit warnings. [exec] +1 system test framework. The patch passed system test framework compile. === Finised Build === > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.1.patch, HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13103297#comment-13103297 ] Uma Maheswara Rao G commented on HDFS-2271: --- Konstantin, Can you please take a look? > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug > Components: name-node >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13099192#comment-13099192 ] Hadoop QA commented on HDFS-2271: - -1 overall. Here are the results of testing the latest attachment http://issues.apache.org/jira/secure/attachment/12493359/HDFS-2271.patch against trunk revision . +1 @author. The patch does not contain any @author tags. +1 tests included. The patch appears to include 3 new or modified tests. -1 patch. The patch command could not apply the patch. Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/1220//console This message is automatically generated. > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > Attachments: HDFS-2271.patch > > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089923#comment-13089923 ] Uma Maheswara Rao G commented on HDFS-2271: --- Hi Konstantin, My code base is from 0.20.1 and back ported the BackupNode :-) In 0.22 also same code is available. Code snippet i pasted above is from 0.22. Logs are from my production systems.(0.20.1 code base). > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089882#comment-13089882 ] Konstantin Shvachko commented on HDFS-2271: --- I am wondering what are you running? BackupNode was introduced when LV was -19, you seem to have -18. Can you reproduce it on say 0.22? > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[ https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087190#comment-13087190 ] Uma Maheswara Rao G commented on HDFS-2271: --- Hi Suresh, Thanks for the feedback...Updated it. --Thanks > startJournalSpool should invoke ProcessIOError with failed storage > directories if createEditLogFile throws any exception. > --- > > Key: HDFS-2271 > URL: https://issues.apache.org/jira/browse/HDFS-2271 > Project: Hadoop HDFS > Issue Type: Bug >Affects Versions: 0.22.0 >Reporter: Uma Maheswara Rao G >Assignee: Uma Maheswara Rao G > > Even If createEditsLogFile failes in startJournalSpool of BackUpStorage, > BackUPNode will proceed with exceptions. -- This message is automatically generated by JIRA. For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2271) startJournalSpool should invoke ProcessIOError with failed storage directories if createEditLogFile throws any exception.
[
https://issues.apache.org/jira/browse/HDFS-2271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13087123#comment-13087123
]
Suresh Srinivas commented on HDFS-2271:
---
Uma, can you please keep the description brief and add the details description
to a comment to avoid lengthy emails.
> startJournalSpool should invoke ProcessIOError with failed storage
> directories if createEditLogFile throws any exception.
> ---
>
> Key: HDFS-2271
> URL: https://issues.apache.org/jira/browse/HDFS-2271
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 0.22.0
>Reporter: Uma Maheswara Rao G
>Assignee: Uma Maheswara Rao G
>
> If createEditsLogFile failes in startJournalSpool of BackUpStorage,
> BackUPNode will proceed with following exceptions.
> 2011-04-09 21:24:43,892 ERROR namenode.FSNamesystem
> (FSEditLog.java:logSync(1207)) - Unable to sync edit log.
> org.apache.hadoop.ipc.RemoteException: java.io.FileNotFoundException:
> /hdfsdata/7/patch/name/data/current/edits (No such file or directory)
> at java.io.RandomAccessFile.open(Native Method)
> at java.io.RandomAccessFile.(RandomAccessFile.java:212)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog$EditLogFileOutputStream.(FSEditLog.java:150)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.createEditLogFile(FSEditLog.java:427)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupStorage.startJournalSpool(BackupStorage.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.BackupNode.journal(BackupNode.java:327)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099)
> at org.apache.hadoop.ipc.Client.call(Client.java:953)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:239)
> at $Proxy4.journal(Unknown Source)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.send(EditLogBackupOutputStream.java:165)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogBackupOutputStream.flushAndSync(EditLogBackupOutputStream.java:145)
> at
> org.apache.hadoop.hdfs.server.namenode.EditLogOutputStream.flush(EditLogOutputStream.java:83)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.logSync(FSEditLog.java:1204)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.startCheckpoint(FSNamesystem.java:5164)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNode.startCheckpoint(NameNode.java:547)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:541)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1105)
> at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1101)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:396)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1099)
> 2011-04-09 21:24:43,893 ERROR namenode.FSNamesystem
> (FSEditLog.java:processIOError(557)) - Unable to log edits to
> 10.18.53.203:50100
> 2011-04-09 21:24:44,353 WARN mortbay.log (Slf4jLog.java:warn(76)) -
> /getimage: java.io.IOException: GetImage failed. java.io.IOException:
> Inconsistent checkpoint fields.
> LV = -18 namespaceID = 2007855034 cTime = 0; checkpointTime = 1302380284708.
> Expecting respectively: -18; 2007855034; 0; 1302380284709
> at
> org.apache.hadoop.hdfs.server.namenode.CheckpointSignature.validateStorageInfo(CheckpointSignature.java:70)
> at
> org.apache.hadoop.hdfs.server.namenode.FSImage.validateCheckpointUpload(FSImage.java:1732)
> at
> org.apache.hadoop.hdfs.server.namenode.GetImageServlet.doGet(GetImageServlet.java:62)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:707)
> at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
> Solution:
> When it failes to createEditLogFile, we can collect all the failed
> directories and can call processIOError with that storageDirectories list
> {code}
> synchronized void startJournalSpool(NamenodeRegistration nnReg)
> throws IOException {
> ..
>
