[
https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114839#comment-14114839
]
Jerry He commented on HBASE-11772:
----------------------------------
Hi, [~ram_krish], [~anoop.hbase]
I did a little more testing for TestHRegionServerBulkLoad. It seems to be
related to the changes in HBASE-11591.
Here is what I found out:
TestHRegionServerBulkLoad does not add BULKLOAD_TIME_KEY to the bulk load
hfiles. But currently we use BULKLOAD_TIME_KEY to determine isBulkLoadResult().
I added BULKLOAD_TIME_KEY to the bulk load hfiles by adding:
{code}
writer.appendFileInfo(StoreFile.BULKLOAD_TIME_KEY,
Bytes.toBytes(System.currentTimeMillis()));
{code}
to TestHRegionServerBulkLoad.createHFile().
Then the test passes without HBASE-11591, but fails with HBASE-11591.
Narrowing down a little deeper, the changes that cause the failure are in
StoreFileScanner.java from HBASE-11591's patch:
{code}
+ if (hasMVCCInfo || this.reader.isBulkLoaded()) {
skipKVsNewerThanReadpoint();
{code}
{code}
+ if (!hasMVCCInfo && this.reader.isBulkLoaded()) {
+ return skipKVsNewerThanReadpoint();
{code}
If I back-out the above changes related for skipKVsNewerThanReadpoint(), then
the test passes fine even with BULKLOAD_TIME_KEY added in the test.
> Bulk load mvcc and seqId issues with native hfiles
> --------------------------------------------------
>
> Key: HBASE-11772
> URL: https://issues.apache.org/jira/browse/HBASE-11772
> Project: HBase
> Issue Type: Bug
> Affects Versions: 0.98.5
> Reporter: Jerry He
> Assignee: Jerry He
> Priority: Critical
> Fix For: 0.99.0, 1.0.0, 2.0.0, 0.98.7
>
> Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch
>
>
> There are mvcc and seqId issues when bulk load native hfiles -- meaning
> hfiles that are direct file copy-out from hbase, not from HFileOutputFormat
> job.
> There are differences between these two types of hfiles.
> Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero
> mvcc values in cells.
> Native hfiles also have MAX_SEQ_ID_KEY.
> Native hfiles do not have BULKLOAD_TIME_KEY.
> Here are a couple of problems I observed when bulk load native hfiles.
> 1. Cells in newly bulk loaded hfiles can be invisible to scan.
> It is easy to re-create.
> Bulk load a native hfile that has a larger mvcc value in cells, e.g 10
> If the current readpoint when initiating a scan is less than 10, the cells in
> the new hfile are skipped, thus become invisible.
> We don't reset the readpoint of a region after bulk load.
> 2. The current StoreFile.isBulkLoadResult() is implemented as:
> {code}
> return metadataMap.containsKey(BULKLOAD_TIME_KEY)
> {code}
> which does not detect bulkloaded native hfiles.
> 3. Another observed problem is possible data loss during log recovery.
> It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create
> steps from HBASE-10958.
> 1) Create an empty table
> 2) Put one row in it (let's say it gets seqid 1)
> 3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile
> can be obtained by copying out from existing table.
> 4) Kill the region server that holds the table's region.
> Scan the table once the region is made available again. The first row, at
> seqid 1, will be missing since the HFile with seqid 100 makes us believe that
> everything that came before it was flushed.
> The problem 3 is probably related to 2. We will be ok if we get the appended
> seqId during bulk load instead of 100 from inside the file.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
