[ https://issues.apache.org/jira/browse/HBASE-11772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14123058#comment-14123058 ]
Jerry He commented on HBASE-11772: ---------------------------------- Hi, [~anoop.hbase] Yes. You explained it! Re-based with attachment 'master-v2' > Bulk load mvcc and seqId issues with native hfiles > -------------------------------------------------- > > Key: HBASE-11772 > URL: https://issues.apache.org/jira/browse/HBASE-11772 > Project: HBase > Issue Type: Bug > Affects Versions: 0.98.5 > Reporter: Jerry He > Assignee: Jerry He > Priority: Critical > Fix For: 0.99.0, 1.0.0, 2.0.0, 0.98.7 > > Attachments: HBASE-11772-0.98.patch, HBASE-11772-master-v1.patch, > HBASE-11772-master-v2.patch > > > There are mvcc and seqId issues when bulk load native hfiles -- meaning > hfiles that are direct file copy-out from hbase, not from HFileOutputFormat > job. > There are differences between these two types of hfiles. > Native hfiles have possible non-zero MAX_MEMSTORE_TS_KEY value and non-zero > mvcc values in cells. > Native hfiles also have MAX_SEQ_ID_KEY. > Native hfiles do not have BULKLOAD_TIME_KEY. > Here are a couple of problems I observed when bulk load native hfiles. > 1. Cells in newly bulk loaded hfiles can be invisible to scan. > It is easy to re-create. > Bulk load a native hfile that has a larger mvcc value in cells, e.g 10 > If the current readpoint when initiating a scan is less than 10, the cells in > the new hfile are skipped, thus become invisible. > We don't reset the readpoint of a region after bulk load. > 2. The current StoreFile.isBulkLoadResult() is implemented as: > {code} > return metadataMap.containsKey(BULKLOAD_TIME_KEY) > {code} > which does not detect bulkloaded native hfiles. > 3. Another observed problem is possible data loss during log recovery. > It is similar to HBASE-10958 reported by [~jdcryans]. Borrow the re-create > steps from HBASE-10958. > 1) Create an empty table > 2) Put one row in it (let's say it gets seqid 1) > 3) Bulk load one native hfile with large seqId ( e.g. 100). The native hfile > can be obtained by copying out from existing table. > 4) Kill the region server that holds the table's region. > Scan the table once the region is made available again. The first row, at > seqid 1, will be missing since the HFile with seqid 100 makes us believe that > everything that came before it was flushed. > The problem 3 is probably related to 2. We will be ok if we get the appended > seqId during bulk load instead of 100 from inside the file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)