bbeaudreault opened a new pull request #3803: URL: https://github.com/apache/hbase/pull/3803
See https://issues.apache.org/jira/browse/HBASE-26304 for more details. This is part of my work on porting our internal [LocalityHealer](https://issues.apache.org/jira/browse/HBASE-26250) tool to the hbase project. HBase data is stored in HDFS Blocks, and HBase tracks the distribution of those blocks' replicas in the HDFSBlockDistribution objects in each StoreFileInfo. The current assumption is that those blocks never move after an HStore is opened, but this is not true. Even without the LocalityHealer, blocks can move if a datanode is decommissioned or someone inadvertently runs the hdfs balancer or mover. But the LocalityHealer is special in that it makes moves with an effort to _improve_ locality for HBase. Due to the above assumption, HBase cannot take advantage of (or properly reflect) out-of-band locality changes. This affects the balancer, some compaction decisions, any downstream users of ClusterMetrics, and metric reporting. This PR adds a new LocalityRefreshChore which periodically updates the block locations for store files. This PR will not actually improve read performance, just the reporting of locality and balancer decisions. The DFSClient will heal itself because it refetches block locations when replicas are not found where expected. I've also submitted https://github.com/apache/hadoop/pull/3527, which makes that more explicit for those users running modern versions of hadoop. Ideally these metrics would be perfectly tied to underlying DFSInputStream, but that has proven complicated to extract, especially since HBase is designed to work against multiple major versions of HDFS and there is no existing mechanism to do so. I'm open to suggestions though. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
