Emil Kleszcz created HBASE-29554:
------------------------------------
Summary: Corrupted hbase:meta rowkeys cannot be deleted
Key: HBASE-29554
URL: https://issues.apache.org/jira/browse/HBASE-29554
Project: HBase
Issue Type: Task
Affects Versions: 2.5.10
Reporter: Emil Kleszcz
*Context:*
It is possible to corrupt the {{hbase:meta}} table by inserting rowkeys
containing a single comma using commands like:
{code:java}
put 'hbase:meta', 'rowkey,', 'info:regioninfo', 'test'{code}
Example of resulting rows in `{{{}hbase:meta`{}}}:
{code:java}
ROW COLUMN+CELL
, column=info:regioninfo,
timestamp=2025-08-26T21:49:33.427, value=
rowkey, column=info:regioninfo,
timestamp=2025-08-26T22:05:52.653, value=
rowkey, column=info:regioninfo,
timestamp=2025-08-26T21:36:56.560, value=
rowkey,a column=info:regioninfo,
timestamp=2025-08-26T22:29:06.576, value= {code}
*Problem:*
* When a rowkey in `{{{}hbase:meta`{}}} contains a single comma, scanners
return the rowkeys correctly, but {{{}get{}}}, {{{}delete{}}}, and
{{deleteall}} operations {*}do not work on these rowkeys{*}.
* This behavior is specific to {{hbase:meta}} and does not occur in user
tables.
*Attempted Workarounds:*
Tried deleting the corrupted row using the Java HBase client:
{code:java}
import org.apache.hadoop.hbase.client.ConnectionFactory
import org.apache.hadoop.hbase.client.Table
import org.apache.hadoop.hbase.TableName
import org.apache.hadoop.hbase.client.Delete
import org.apache.hadoop.hbase.util.Bytes
connection = ConnectionFactory.createConnection
tabletable = connection.getTable(TableName.valueOf("hbase:meta"))
keyrowKey = Bytes.toBytes("rowkey,")
delete = Delete.new(rowKey)
delete.setTimestamp(Long::MAX_VALUE)
delete.addColumn(Bytes.toBytes("info"), Bytes.toBytes("state"))
table.delete(delete)
table.close
connection.close{code}
*Result:* Unable to delete the corrupted row.
It is possible to insert a properly serialized region info for such keys:
{code:java}
rowKey = Bytes.toBytes("rowkey,")
tableName = TableName.valueOf("TEST:TEST")
Infobuilder = RegionInfoBuilder.newBuilder(tableName)
builder.setStartKey(Bytes.toBytes(""))
builder.setEndKey(Bytes.toBytes("1"))
builder.setRegionId(12345)
fakeRegion = builder.build()
serializedValue = RegionInfo.toByteArray(fakeRegion)
put = Put.new(rowKey)put.addColumn(HConstants::CATALOG_FAMILY,
HConstants::REGIONINFO_QUALIFIER, serializedValue) {code}
Resulting row in {{{}hbase:meta{}}}:
{code:java}
rowkey, column=info:regioninfo,
timestamp=2025-08-27T17:02:21.565, value={ENCODED =>
a6839d33e016dc75cfb9ac9c74a576c8, NAME => 'TEST:TEST,,1.a6839d33e0
16dc75cfb9ac9c74a576c8.',
STARTKEY => '', ENDKEY => '1'} {code}
*Problems / Risks:*
# Corrupted rowkeys with commas in {{hbase:meta}} cannot be fixed or removed
on a running cluster.
# Allowing insertion of such rowkeys is risky. If insertion is allowed, there
must be a mechanism for administrators to clean them up.
# {{HBCK2}} does not resolve the issue (e.g., {{fixMeta}} fails).
*Request / Suggested Action:*
* Investigate why rowkeys with a single comma break deletion in
{{{}hbase:meta{}}}.
* Consider adding safeguards to prevent insertion of rowkeys with wrong format
into {{{}hbase:meta{}}}.
* Provide an administrative method to safely remove or repair corrupted meta
rowkeys.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
