[
https://issues.apache.org/jira/browse/HADOOP-1520?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12507565
]
dhruba borthakur commented on HADOOP-1520:
------------------------------------------
The DFSClient retries the ClientProtocol.complete() call untill it is
successful. I am assuming that a previous complete() call was successfully
executed at the NameNode but the client decided to declare that call as
"timedout" and retried. The retried call experiences this Exception over and
over again.
One generic solution is a have a unique sequence number per RPC that identifies
that RPC. The NameNode remembers the last few thousands RPCs that it executed.
If a client sends a RPC with a sequence number that exists in the NameNode
cache, the NameNode returns the results that were produced by the previous
execution of the same RPC.
Can you pl send a pointer to a namenode log file that shows this problem?
> IndexOutOfBoundsException in FSEditLog.processIOError
> -----------------------------------------------------
>
> Key: HADOOP-1520
> URL: https://issues.apache.org/jira/browse/HADOOP-1520
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Affects Versions: 0.14.0
> Reporter: Nigel Daley
> Assignee: Konstantin Shvachko
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Running NNBench I saw these exceptions in the NameNode logs:
> NAMENODE:
> 2007-06-21 04:02:53,587 INFO org.apache.hadoop.fs.FSNamesystem: Roll FSImage
> 2007-06-21 04:02:53,594 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 0 on 8020, call complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc,
> DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException:
> java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> java.io.IOException: java.lang.IndexOutOfBoundsException: Index: 0, Size: 0
> at java.util.ArrayList.RangeCheck(ArrayList.java:547)
> at java.util.ArrayList.remove(ArrayList.java:387)
> at org.apache.hadoop.dfs.FSEditLog.processIOError(FSEditLog.java:169)
> at org.apache.hadoop.dfs.FSEditLog.logSync(FSEditLog.java:407)
> at
> org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:878)
> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:346)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> ...
> 2007-06-21 04:02:53,823 WARN org.apache.hadoop.dfs.StateChange: DIR*
> NameSystem.completeFile: failed to complete
> /user/hadoopqa/nameNode100Benchmark/output/.214.crc because
> dir.getFileBlocks() is non-null and pendingFile is null
> 2007-06-21 04:02:53,828 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 14 on 8020, call
> complete(/user/hadoopqa/nameNode100Benchmark/output/.214.crc,
> DFSClient_-1646448212) from 72.30.51.75:56245: error: java.io.IOException:
> Could not complete write to file
> /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> java.io.IOException: Could not complete write to file
> /user/hadoopqa/nameNode100Benchmark/output/.214.crc by DFSClient_-1646448212
> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:352)
> at sun.reflect.GeneratedMethodAccessor16.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> This last exception occurs over and over again forever (this could be a
> result of the way NNBench is written).
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
