[
https://issues.apache.org/jira/browse/HBASE-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12580922#action_12580922
]
stack commented on HBASE-534:
-----------------------------
So, in the above exception, the complaint is about a file that was in the
parent of this daughter region.
Looking back on the opening of this daughter region, I see following exceptions:
{code}
2008-03-16 14:12:40,624 DEBUG org.apache.hadoop.hbase.HStore: started
compaction of 2 files using
hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/compaction.dir for
1663054471/contents2008-03-16 14:12:48,142 INFO org.apache.hadoop.fs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.dfs.LeaseExpiredException: No lease on
/hbase/pdc-docs/compaction.dir/1663054471/contents/mapfiles/5227279456974119035/data
File does not exist. [Lease. Holder: 44 46 53 43 6c 69 65 6e 74 5f 32 30 39
32 36 34 37 35 33 36, heldlocks: 0, pendingcreates: 3]
at org.apache.hadoop.dfs.FSNamesystem.checkLease(FSNamesystem.java:1160)
at
org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1097)
at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:312)
at sun.reflect.GeneratedMethodAccessor15.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:409)
at org.apache.hadoop.ipc.Server$Handler.run(Server.java:899)
at org.apache.hadoop.ipc.Client.call(Client.java:512)
at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:198)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at sun.reflect.GeneratedMethodAccessor8.invoke(Unknown Source)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:585)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
at
org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
at org.apache.hadoop.dfs.$Proxy1.addBlock(Unknown Source)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2061)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:1954)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1500(DFSClient.java:1479)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1593)
... above exception then is retried:
2008-03-16 14:12:48,143 WARN org.apache.hadoop.fs.DFSClient:
NotReplicatedYetException sleeping
/hbase/pdc-docs/compaction.dir/1663054471/contents/mapfiles/5227279456974119035/data
retries left 4
...
Until
2008-03-16 14:12:54,196 ERROR org.apache.hadoop.hbase.HRegionServer: Compaction
failed for region pdc-docs,EP05011250NWA2,1205705554585
java.io.IOException: Could not get block locations. Aborting... at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.processDatanodeError(DFSClient.java:1824)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1100(DFSClient.java:1479)
at
org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1571)
{code}
Perhaps it was the second file of the two files to compact?
Region goes on to open? 30 minutes later we try to compact again but fails
with:
{code}
2008-03-16 14:42:40,726 ERROR org.apache.hadoop.hbase.HRegionServer: Compaction
failed for region pdc-docs,EP05011250NWA2,1205705554585
java.io.FileNotFoundException: File
hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/101719651524869690/data
does not exist.
{code}
Subsequently, the region continues to 'work' but after every flush tries to
compact and fails with:
{code}
2008-03-17 08:42:41,380 WARN org.apache.hadoop.hbase.HStore: Failed with
java.io.FileNotFoundException: File
hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/101719651524869690/data
does not exist.:
1663054471/contents/5669005871484475450/1733592281/101719651524869690/bottom
1733592281/101719651524869690/bottom for 1663054471/contents
2008-03-17 08:42:41,422 ERROR org.apache.hadoop.hbase.HRegionServer: Compaction
failed for region pdc-docs,EP05011250NWA2,1205705554585
java.io.FileNotFoundException: File
hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/101719651524869690/data
does not exist.
at
org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1387)
at
org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1382)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
at
org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:600)
at
org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:655)
at
org.apache.hadoop.hbase.HStoreFile$HalfMapFileReader.<init>(HStoreFile.java:758)
at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
at org.apache.hadoop.hbase.HStore.compactHStoreFiles(HStore.java:1281)
at org.apache.hadoop.hbase.HStore.compact(HStore.java:1245)
at org.apache.hadoop.hbase.HRegion.compactStores(HRegion.java:793)
at org.apache.hadoop.hbase.HRegion.compactIfNeeded(HRegion.java:725)
at
org.apache.hadoop.hbase.HRegionServer$CompactSplitThread.run(HRegionServer.java:254)
{code}
This seems like a simple enough exception to handle... should at least remove
the Reader that is throwing the FNF from our list of ready-Readers.
> Stores retaining references to long-deleted mapfiles
> ----------------------------------------------------
>
> Key: HBASE-534
> URL: https://issues.apache.org/jira/browse/HBASE-534
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.1.0
> Reporter: stack
>
> Saw the following on the Lars clusters (He's up on 0.16.1 and very recent 0.1
> branch) trying to run a scan over all his content:
> {code}
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2241)
> at
> org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java:2362)
> at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2152)
> at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java:1640)
> at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1214)
> at
> org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1448)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> Caused by: java.io.FileNotFoundException: File
> hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/3435064940161142159/data
> does not exist.
> at
> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1387)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1382)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
> at
> org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:600)
> at
> org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:655)
> at
> org.apache.hadoop.hbase.HStoreFile$HalfMapFileReader.<init>(HStoreFile.java:758)
> at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
> at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2216)
> ... 11 more
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.HTable$ClientScanner.nextScanner(HTable.java:874)
> at org.apache.hadoop.hbase.HTable$ClientScanner.next(HTable.java:915)
> at
> org.apache.hadoop.hbase.hql.SelectCommand.scanPrint(SelectCommand.java:233)
> at
> org.apache.hadoop.hbase.hql.SelectCommand.execute(SelectCommand.java:100)
> at
> org.apache.hadoop.hbase.hql.HQLClient.executeQuery(HQLClient.java:50)
> at org.apache.hadoop.hbase.Shell.main(Shell.java:114)
> {code}
> The scanner breaks when it hits the above exception. The odd thing is that
> the referenced mapfile is out of a region that was deleted 4 days ago after
> purportedly all references had been let go:
> {code}
> 2008-03-16 15:13:36,744 DEBUG org.apache.hadoop.hbase.HRegion: DELETING
> region hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
