[
https://issues.apache.org/jira/browse/HBASE-10193?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13853077#comment-13853077
]
Aditya Kishore commented on HBASE-10193:
----------------------------------------
I thought of that but there is already a sufficient amount of log message when
such an even happen. Here is a sample.
{noformat}
2013-12-19 09:57:25,231 INFO org.apache.hadoop.hbase.regionserver.Store: Closed
f1
2013-12-19 09:57:25,232 INFO org.apache.hadoop.hbase.regionserver.Store: Closed
f2
2013-12-19 09:57:25,233 ERROR
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Failed open of
region=t1,,1387325742703.fe9d65589414b96187bab560f74b18fa., starting to roll
back the global memstore size.
java.io.IOException: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
/hbase/t1/fe9d65589414b96187bab560f74b18fa/f3/badc765c02904212b94f6f6e91f290a0
at
org.apache.hadoop.hbase.regionserver.HRegion.initializeRegionInternals(HRegion.java:629)
at
org.apache.hadoop.hbase.regionserver.HRegion.initialize(HRegion.java:548)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4474)
at
org.apache.hadoop.hbase.regionserver.HRegion.openHRegion(HRegion.java:4422)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.openRegion(OpenRegionHandler.java:333)
at
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler.process(OpenRegionHandler.java:101)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:175)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
Caused by: java.io.IOException:
org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem reading HFile
Trailer from file
/hbase/t1/fe9d65589414b96187bab560f74b18fa/f3/badc765c02904212b94f6f6e91f290a0
at
org.apache.hadoop.hbase.regionserver.Store.loadStoreFiles(Store.java:483)
at org.apache.hadoop.hbase.regionserver.Store.<init>(Store.java:260)
at
org.apache.hadoop.hbase.regionserver.HRegion.instantiateHStore(HRegion.java:3384)
at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:601)
at org.apache.hadoop.hbase.regionserver.HRegion$2.call(HRegion.java:1)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
... 3 more
Caused by: org.apache.hadoop.hbase.io.hfile.CorruptHFileException: Problem
reading HFile Trailer from file
/hbase/t1/fe9d65589414b96187bab560f74b18fa/f3/badc765c02904212b94f6f6e91f290a0
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:582)
at
org.apache.hadoop.hbase.io.hfile.HFile.createReaderWithEncoding(HFile.java:615)
at
org.apache.hadoop.hbase.regionserver.StoreFile$Reader.<init>(StoreFile.java:1368)
at
org.apache.hadoop.hbase.regionserver.StoreFile.open(StoreFile.java:558)
at
org.apache.hadoop.hbase.regionserver.StoreFile.createReader(StoreFile.java:666)
at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:459)
at org.apache.hadoop.hbase.regionserver.Store$1.call(Store.java:1)
... 8 more
Caused by: java.lang.IllegalArgumentException: Invalid HFile version: 0
(expected to be between 1 and 2)
at
org.apache.hadoop.hbase.io.hfile.HFile.checkFormatVersion(HFile.java:791)
at
org.apache.hadoop.hbase.io.hfile.FixedFileTrailer.readFromStream(FixedFileTrailer.java:323)
at
org.apache.hadoop.hbase.io.hfile.HFile.pickReaderVersion(HFile.java:551)
... 14 more
2013-12-19 09:57:25,233 INFO
org.apache.hadoop.hbase.regionserver.handler.OpenRegionHandler: Opening of
region {NAME => 't1,,1387325742703.fe9d65589414b96187bab560f74b18fa.', STARTKEY
=> '', ENDKEY => '', ENCODED => fe9d65589414b96187bab560f74b18fa,} failed,
marking as FAILED_OPEN in ZK
{noformat}
> Cleanup HRegion if one of the store fails to open at region initialization
> --------------------------------------------------------------------------
>
> Key: HBASE-10193
> URL: https://issues.apache.org/jira/browse/HBASE-10193
> Project: HBase
> Issue Type: Bug
> Components: regionserver
> Affects Versions: 0.96.1, 0.94.14
> Reporter: Aditya Kishore
> Assignee: Aditya Kishore
> Priority: Critical
> Attachments: HBASE-10193.patch, HBASE-10193_0.94.patch,
> HBASE-10193_0.94_v2.patch, HBASE-10193_0.94_v3.patch, HBASE-10193_v2.patch,
> HBASE-10193_v3.patch
>
>
> While investigating a different issue, I realized that the fix for HBASE-9737
> is not sufficient to prevent resource leak if a region fails to open for some
> reason, say a corrupt HFile.
> The region may have, by then, opened other good HFiles in that store or other
> stores if it has more than one column family and their streams may leak if
> not closed.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
