[
https://issues.apache.org/jira/browse/HADOOP-1765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#action_12521985
]
stack commented on HADOOP-1765:
-------------------------------
Here are all logs related to failing file pulled from attached DEBUG log:
{code}
2007-08-22 23:48:59,515 DEBUG org.apache.hadoop.dfs.StateChange: *DIR*
NameNode.create: file
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-2109891694 at 208.76.44.140
2007-08-22 23:48:59,515 DEBUG org.apache.hadoop.dfs.StateChange: DIR*
NameSystem.startFile: file
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-2109891694 at 208.76.44.1402007-08-22 23:48:59,515 DEBUG
org.apache.hadoop.dfs.StateChange: DIR* NameSystem.startFile: add
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
to pendingCreates for DFSClient_-2109891694
2007-08-22 23:48:59,515 DEBUG org.apache.hadoop.dfs.StateChange: DIR*
FSDirectory.addFile:
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
with 0 blocks is added to the file system
2007-08-22 23:49:00,811 DEBUG org.apache.hadoop.dfs.StateChange: *BLOCK*
NameNode.addBlock: file
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-2109891694
2007-08-22 23:49:00,811 DEBUG org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.getAdditionalBlock: file
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-2109891694
2007-08-22 23:49:00,812 INFO org.apache.hadoop.dfs.StateChange: BLOCK*
NameSystem.allocateBlock:
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data.
blk_-7962476260520498222 is created and added to pendingCreates and
pendingCreateBlocks
2007-08-22 23:49:03,352 DEBUG org.apache.hadoop.dfs.StateChange: *DIR*
NameNode.complete:
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-21098916942007-08-22 23:49:03,352 DEBUG
org.apache.hadoop.dfs.StateChange: DIR* NameSystem.completeFile:
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
for DFSClient_-2109891694
2007-08-22 23:49:03,354 INFO org.apache.hadoop.ipc.Server: IPC Server handler 5
on 9000, call
complete(/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data,
DFSClient_-2109891694) from 208.76.44.140:56731: error: java.io.IOException:
Unknown file:
/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,F2pGuwUpH4RWZY5e1ZqKjF==,8020883803428152097/repo/mapfiles/-1/data
{code}
Looks like a block is added to a file known by the filesystem, then another,
but then on 'complete', IOE.
> IOException on close: Unknown file
> ----------------------------------
>
> Key: HADOOP-1765
> URL: https://issues.apache.org/jira/browse/HADOOP-1765
> Project: Hadoop
> Issue Type: Bug
> Components: dfs
> Reporter: stack
> Priority: Minor
> Attachments: namenode-excerpt.log.gz
>
>
> On occasion I get an IOE trying to close a MapFile I've just opened and
> filled. Below is the content of the namenode log. The first thing in it
> after startup messages is the IOE followed by other messages about
> '...current leaseholder is trying to recreate file'. It repeats for ever.
> This is TRUNK r567876.
> I filed this issue at Raghu's suggestion. Here's what he said on the list:
> .bq Yes, the earler message was also from Namenode log. Before that line
> there should have been a message that adds a block to this file in the same
> log. I think you should file a Jira for this. It probably related to
> HADOOP-999. Raghu.
> {code}
> 2007-08-22 21:38:49,071 INFO org.apache.hadoop.net.NetworkTopology: Adding a
> new node: /default-rack/208.76.44.139:50010
> 2007-08-22 21:43:50,216 INFO org.apache.hadoop.fs.FSNamesystem: Roll Edit Log
> from 208.76.44.139
> 2007-08-22 21:45:21,459 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 9000, call
> complete(/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data,
> DFSClient_1857293290) from 208.76.44.139:52301: error: java.io.IOException:
> Unknown file:
> /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
> java.io.IOException: Unknown file:
> /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
> at org.apache.hadoop.dfs.FSDirectory.addBlocks(FSDirectory.java:561)
> at
> org.apache.hadoop.dfs.FSNamesystem.completeFileInternal(FSNamesystem.java:1002)
> at org.apache.hadoop.dfs.FSNamesystem.completeFile(FSNamesystem.java:952)
> at org.apache.hadoop.dfs.NameNode.complete(NameNode.java:348)
> at sun.reflect.GeneratedMethodAccessor10.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> 2007-08-22 21:45:24,394 WARN org.apache.hadoop.dfs.StateChange: DIR*
> NameSystem.startFile: failed to create file
> /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
> for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder
> is trying to recreate file.
> 2007-08-22 21:45:24,394 INFO org.apache.hadoop.ipc.Server: IPC Server handler
> 1 on 9000, call
> create(/bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data,
> DFSClient_1857293290, true, 3, 67108864) from 208.76.44.139:52312: error:
> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create file
> /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
> for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder
> is trying to recreate file.
> org.apache.hadoop.dfs.AlreadyBeingCreatedException: failed to create file
> /bfd/hadoop-stack-data/tmp/hbase/compaction.tmp/hregion_hbaserepository,,8918388410463499185/repo/mapfiles/-1/data
> for DFSClient_1857293290 on client 208.76.44.139 because current leaseholder
> is trying to recreate file.
> at org.apache.hadoop.dfs.FSNamesystem.startFile(FSNamesystem.java:740)
> at org.apache.hadoop.dfs.NameNode.create(NameNode.java:307)
> at sun.reflect.GeneratedMethodAccessor7.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:340)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:566)
> {code}
> Let me try and replicate with DEBUG level enabled. Also, I think I know how
> to easily replicate. Need to do some tests.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
