[
https://issues.apache.org/jira/browse/HBASE-24189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17134870#comment-17134870
]
Anoop Sam John commented on HBASE-24189:
----------------------------------------
Pushed to branch-1 also. Thanks all for the reviews.
> WALSplit recreates region dirs for deleted table with recovered edits data
> --------------------------------------------------------------------------
>
> Key: HBASE-24189
> URL: https://issues.apache.org/jira/browse/HBASE-24189
> Project: HBase
> Issue Type: Bug
> Components: regionserver, wal
> Affects Versions: 2.2.4
> Environment: * HDFS 3.1.3
> * HBase 2.1.4
> * OpenJDK 8
> Reporter: Andrey Elenskiy
> Assignee: Anoop Sam John
> Priority: Major
> Fix For: 3.0.0-alpha-1, 2.3.0, 1.7.0, 2.2.6
>
>
> Under the following scenario region directories in HDFS can be recreated with
> only recovered.edits in them:
> # Create table "test"
> # Put into "test"
> # Delete table "test"
> # Create table "test" again
> # Crash the regionserver to which the put has went to force the WAL replay
> # Region directory in old table is recreated in new table
> # hbase hbck returns inconsistency
> This appears to happen due to the fact that WALs are not cleaned up once a
> table is deleted and they still contain the edits from old table. I've tried
> wal_roll command on the regionserver before crashing it, but it doesn't seem
> to help as under some circumstances there are still WAL files around. The
> only solution that works consistently is to restart regionserver before
> creating the table at step 4 because that triggers log cleanup on startup:
> [https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508|https://github.com/apache/hbase/blob/f3ee9b8aa37dd30d34ff54cd39fb9b4b6d22e683/hbase-procedure/src/main/java/org/apache/hadoop/hbase/procedure2/store/wal/WALProcedureStore.java#L508)]
>
> Truncating a table also would be a workaround by in our case it's a no-go as
> we create and delete tables in our tests which run back to back (create table
> in the beginning of the test and delete in the end of the test).
> A nice option in our case would be to provide hbase shell utility to force
> clean up of log files manually as I realize that it's not really viable to
> clean all of those up every time some table is removed.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
