[
https://issues.apache.org/jira/browse/HBASE-24920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17188033#comment-17188033
]
Andrey Elenskiy commented on HBASE-24920:
-----------------------------------------
Was away for a week and just attempted to write a prototype. The tool is fairly
simple: if HFileScanner.next() throws exception, search for DATABLK magic
offset in FSInputStream and resume from there.
However, I realized that it wouldn't be possible to have it be part of hbase
operator tools. I rely on some private classes in
org.apache.hadoop.hbase.io.hfile package which change quite a bit between
versions (2.2.5, 2.3.0, master are all different) making it impossible as a
drop in jar. So, there are couple of choices here:
1. package the tool with dependencies and pin to some hbase version (which one?)
2. package along with hbase being part of "hbase hfile" command (probably why
[~stack] suggested it in the first place ;))
3. completely re-implement needed pieces in hbase operator tool so that there
are no dependencies on hbase version (making this tool hard to maintain).
> A tool to rewrite corrupted HFiles
> ----------------------------------
>
> Key: HBASE-24920
> URL: https://issues.apache.org/jira/browse/HBASE-24920
> Project: HBase
> Issue Type: Brainstorming
> Components: hbase-operator-tools
> Reporter: Andrey Elenskiy
> Priority: Major
>
> Typically I have been dealing with corrupted HFiles (due to loss of hdfs
> blocks) by just removing them. However, It always seemed wasteful to throw
> away the entire HFile (which can be hundreds of gigabytes), just because one
> hdfs block is missing (128MB).
> I think there's a possibility for a tool that can rewrite an HFile by
> skipping corrupted blocks.
> There can be multiple types of issues with hdfs blocks but any of them can be
> treated as if the block doesn't exist:
> 1. All the replicas can be lost
> 2. The block can be corrupted due to some bug in hdfs (I've recently run into
> HDFS-15186 by experimenting with EC).
> At the simplest the tool can be a local mapreduce job (mapper only) with a
> custom HFile reader input that can seek to next DATABLK to skip corrupted
> hdfs blocks.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
