kadirozde commented on PR #5545: URL: https://github.com/apache/hbase/pull/5545#issuecomment-1998519608
@Apache9, I have moved historical file generation to StoreFileWriter. Based on a new config param called hbase.enable.historical.compaction.files (please suggest a better name if you like), StoreFileWriter uses a single file (just live file) or two file (live vs historical). This new config param replaces the previous one (hbase.hstore.defaultengine.enable.dualfilewriter). I also changed the name of the new store file metadata key. Now it is called HISTORICAL. This is set only when the file is historical. Currently, we generate historical files only for compactions. However, we can enable it for flushes too. We can introduce another config param say base.enable.historical.flush.files to enable it for flushes. It will be a simple change. Please let me know if you like me to include this change too. Since historical file generated at the store file, it is available for all types of store engines and compactions by default. I have not disabled it for specific types of compactions yet. Either we can disable it for them or add tests to cover them. Let me know how you like to proceed. I also introduced a comprehensive data integrity checks with a new integration test class called TestStoreFileWriter. The test identified data integrity issues with version delete markers. I verified that these issues are not introduced by this PR and currently exist in hbase. I will file a jira for this. After your feedback, I will update the design doc and PR. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
