[
https://issues.apache.org/jira/browse/HBASE-534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12581660#action_12581660
]
stack commented on HBASE-534:
-----------------------------
Studying more, root of problem is double assignment of region. Below is master
log excerpts. See how pdc-docs,EP05011250NWA2,1205705554585 is doubly-assigned:
{code}
2008-03-16 15:12:36,626 DEBUG org.apache.hadoop.hbase.HMaster: Current
assignment of pdc-docs,EP05011250NWA2,1205705554585 is not valid: storedInfo:
null, startCode: -1, storedInfo.startCode: -1, unassignedRegions: false,
pendingRegions: false
2008-03-16 15:12:37,463 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP05011250NWA2,1205705554585 to server 192.168.105.21:60020
2008-03-16 15:12:38,406 INFO org.apache.hadoop.hbase.HMaster: region
pdc-docs,EP05011250NWA2,1205704064309 split. New regions are:
pdc-docs,EP05011250NWA2,1205705554585, pdc-docs,EP05076594NWA2,1205705554586
2008-03-16 15:12:38,499 INFO org.apache.hadoop.hbase.HMaster: assigning region
pdc-docs,EP05011250NWA2,1205705554585 to server 192.168.105.39:60020
2008-03-16 15:12:40,490 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REPORT_PROCESS_OPEN : pdc-docs,EP05011250NWA2,1205705554585 from
192.168.105.21:60020
2008-03-16 15:12:40,490 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REPORT_OPEN : pdc-docs,EP05011250NWA2,1205705554585 from
192.168.105.21:60020
2008-03-16 15:12:40,490 INFO org.apache.hadoop.hbase.HMaster:
192.168.105.21:60020 serving pdc-docs,EP05011250NWA2,1205705554585
2008-03-16 15:12:40,491 INFO org.apache.hadoop.hbase.HMaster: regionname:
pdc-docs,EP05011250NWA2,1205705554585, startKey: <EP05011250NWA2>, endKey:
<EP05076594NWA2>, encodedName: 1663054471, tableDesc: {name: pdc-docs,
families: {contents:={name: contents, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, language:={name:
language, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, mimetype:={name: mimetype, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}}} open on 192.168.105.21:60020
2008-03-16 15:12:40,491 INFO org.apache.hadoop.hbase.HMaster: updating row
pdc-docs,EP05011250NWA2,1205705554585 in table .META.,,1 with startcode
1205660242301 and server 192.168.105.21:60020
2008-03-16 15:12:41,530 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REPORT_PROCESS_OPEN : pdc-docs,EP05011250NWA2,1205705554585 from
192.168.105.39:60020
2008-03-16 15:12:41,530 DEBUG org.apache.hadoop.hbase.HMaster: Received
MSG_REPORT_OPEN : pdc-docs,EP05011250NWA2,1205705554585 from
192.168.105.39:60020
2008-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster:
192.168.105.39:60020 serving pdc-docs,EP05011250NWA2,1205705554585
2008-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster: regionname:
pdc-docs,EP05011250NWA2,1205705554585, startKey: <EP05011250NWA2>, endKey:
<EP05076594NWA2>, encodedName: 1663054471, tableDesc: {name: pdc-docs,
families: {contents:={name: contents, max versions: 3, compression: NONE, in
memory: false, max length: 2147483647, bloom filter: none}, language:={name:
language, max versions: 3, compression: NONE, in memory: false, max length:
2147483647, bloom filter: none}, mimetype:={name: mimetype, max versions: 3,
compression: NONE, in memory: false, max length: 2147483647, bloom filter:
none}}} open on 192.168.105.39:60020
2006-03-16 15:12:41,530 INFO org.apache.hadoop.hbase.HMaster: updating row
pdc-docs,EP05011250NWA2,1205705554585 in table .META.,,1 with startcode
1205660242518 and server 192.168.105.39:60020
2008-03-16 15:13:36,741 DEBUG org.apache.hadoop.hbase.HMaster:
pdc-docs,EP05011250NWA2,1205705554585 no longer has references to
pdc-docs,EP05011250NWA2,1205704064309
{code}
Happening because regionserver added regions to META, scan happened and marked
the regions as needing assignment because server was empty and startcode -1,
regions were added to unassignedRegions and actually assigned... then in came
the SPLIT region message and regions were added again to unassignedRegions (no
check in split block if regions already present).
> Stores retaining references to long-deleted mapfiles
> ----------------------------------------------------
>
> Key: HBASE-534
> URL: https://issues.apache.org/jira/browse/HBASE-534
> Project: Hadoop HBase
> Issue Type: Bug
> Affects Versions: 0.1.0
> Reporter: stack
> Priority: Blocker
> Fix For: 0.1.0
>
>
> Saw the following on the Lars clusters (He's up on 0.16.1 and very recent 0.1
> branch) trying to run a scan over all his content:
> {code}
> java.io.IOException: java.io.IOException: HStoreScanner failed construction
> at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2241)
> at
> org.apache.hadoop.hbase.HStore$HStoreScanner.<init>(HStore.java:2362)
> at org.apache.hadoop.hbase.HStore.getScanner(HStore.java:2152)
> at org.apache.hadoop.hbase.HRegion$HScanner.<init>(HRegion.java:1640)
> at org.apache.hadoop.hbase.HRegion.getScanner(HRegion.java:1214)
> at
> org.apache.hadoop.hbase.HRegionServer.openScanner(HRegionServer.java:1448)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:585)
> at org.apache.hadoop.hbase.ipc.HbaseRPC$Server.call(HbaseRPC.java:413)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:910)
> Caused by: java.io.FileNotFoundException: File
> hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281/contents/mapfiles/3435064940161142159/data
> does not exist.
> at
> org.apache.hadoop.dfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:341)
> at org.apache.hadoop.fs.FileSystem.getLength(FileSystem.java:538)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1387)
> at
> org.apache.hadoop.io.SequenceFile$Reader.<init>(SequenceFile.java:1382)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:254)
> at org.apache.hadoop.io.MapFile$Reader.<init>(MapFile.java:242)
> at
> org.apache.hadoop.hbase.HStoreFile$HbaseMapFile$HbaseReader.<init>(HStoreFile.java:600)
> at
> org.apache.hadoop.hbase.HStoreFile$BloomFilterMapFile$Reader.<init>(HStoreFile.java:655)
> at
> org.apache.hadoop.hbase.HStoreFile$HalfMapFileReader.<init>(HStoreFile.java:758)
> at org.apache.hadoop.hbase.HStoreFile.getReader(HStoreFile.java:424)
> at
> org.apache.hadoop.hbase.HStore$StoreFileScanner.<init>(HStore.java:2216)
> ... 11 more
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:494)
> at
> org.apache.hadoop.hbase.RemoteExceptionHandler.decodeRemoteException(RemoteExceptionHandler.java:82)
> at
> org.apache.hadoop.hbase.HTable$ClientScanner.nextScanner(HTable.java:874)
> at org.apache.hadoop.hbase.HTable$ClientScanner.next(HTable.java:915)
> at
> org.apache.hadoop.hbase.hql.SelectCommand.scanPrint(SelectCommand.java:233)
> at
> org.apache.hadoop.hbase.hql.SelectCommand.execute(SelectCommand.java:100)
> at
> org.apache.hadoop.hbase.hql.HQLClient.executeQuery(HQLClient.java:50)
> at org.apache.hadoop.hbase.Shell.main(Shell.java:114)
> {code}
> The scanner breaks when it hits the above exception. The odd thing is that
> the referenced mapfile is out of a region that was deleted 4 days ago after
> purportedly all references had been let go:
> {code}
> 2008-03-16 15:13:36,744 DEBUG org.apache.hadoop.hbase.HRegion: DELETING
> region hdfs://lv1-xen-pdc-2.worldlingo.com:9000/hbase/pdc-docs/1733592281
> {code}
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
