[jira] [Resolved] (HBASE-24915) Improve BlockCache read performance by specifying BlockType
[
https://issues.apache.org/jira/browse/HBASE-24915?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-24915.
---
Hadoop Flags: Reviewed
Resolution: Fixed
Added you as a contributor [~fanrui] . Thanks for the patch (Thanks for review
[~vjasani] )
> Improve BlockCache read performance by specifying BlockType
> ---
>
> Key: HBASE-24915
> URL: https://issues.apache.org/jira/browse/HBASE-24915
> Project: HBase
> Issue Type: Improvement
> Components: BlockCache, Performance
>Reporter: fanrui
>Assignee: fanrui
>Priority: Major
>
> CombinedBlockCache contains l1Cache and l2Cache. l1Cache stores MetaBlock and
> l2Cache stores DataBlock. Because getBlock does not know the BlockType, the
> getBlock of CombinedBlockCache queries l1Cache first, and then l2Cache. But
> actually querying DataBlock is not necessary to query l1Cache.
> Therefore, in some cases where BlockType is known, BlockCache read
> performance can be improved.
> h2. Code：
> BlockCache: default call old getBlock
> {code:java}
> default Cacheable getBlock(BlockCacheKey cacheKey, boolean caching, boolean
> repeat,
> boolean updateCacheMetrics, BlockType blockType) {
> return getBlock(cacheKey, caching, repeat, updateCacheMetrics);
> }
> {code}
> CombinedBlockCache:
> {code:java}
> @Override
> public Cacheable getBlock(BlockCacheKey cacheKey, boolean caching, boolean
> repeat,
> boolean updateCacheMetrics, BlockType blockType) {
> if (blockType == null) {
> return getBlock(cacheKey, caching, repeat, updateCacheMetrics);
> }
> boolean metaBlock = isMetaBlock(blockType);
> if (metaBlock) {
> return l1Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
> } else {
> return l2Cache.getBlock(cacheKey, caching, repeat, updateCacheMetrics);
> }
> }private boolean isMetaBlock(BlockType blockType) {
> return blockType.getCategory() != BlockCategory.DATA;
> }
> {code}
> HFileReaderImpl#getCachedBlock call BlockCache#getBlock(XXX,
> expectedBlockType)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24918) Make RegionInfo#UNDEFINED IA.Private
[ https://issues.apache.org/jira/browse/HBASE-24918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Michael Stack resolved HBASE-24918. --- Hadoop Flags: Reviewed Resolution: Fixed Merged PR. Thanks for patience and reivews [~bharathv] , [~vjasani] , and [~zhangduo] > Make RegionInfo#UNDEFINED IA.Private > > > Key: HBASE-24918 > URL: https://issues.apache.org/jira/browse/HBASE-24918 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.4.0, 2.3.2 >Reporter: Michael Stack >Assignee: Michael Stack >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.3.2 > > > From review of parent PR, RI#UNDEFINED is for internal use only... note it w/ > a IA.Private marking. Also deprecate it so we have option to remove as its > loading implicated in the parent issue static load deadlock. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24861) Release 2.3.1
[ https://issues.apache.org/jira/browse/HBASE-24861?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24861. -- Resolution: Done > Release 2.3.1 > - > > Key: HBASE-24861 > URL: https://issues.apache.org/jira/browse/HBASE-24861 > Project: HBase > Issue Type: Task > Components: community >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24868) Send announce email
[ https://issues.apache.org/jira/browse/HBASE-24868?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24868. -- Resolution: Done > Send announce email > --- > > Key: HBASE-24868 > URL: https://issues.apache.org/jira/browse/HBASE-24868 > Project: HBase > Issue Type: Sub-task >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[ANNOUNCE] Apache HBase 2.3.1 is now available for download
The HBase team is happy to announce the immediate availability of HBase 2.3.1. Apache HBase™ is an open-source, distributed, versioned, non-relational database. Apache HBase gives you low latency random access to billions of rows with millions of columns atop non-specialized hardware. To learn more about HBase, see https://hbase.apache.org/. HBase 2.3.1 is the first patch release in the HBase 2.3.x line, which aims to improve the stability and reliability of HBase. This release includes 50 bug fixes and improvements since 2.3.0. The full list of issues can be found in the included CHANGES.md and RELEASENOTES.md, or via our issue tracker: https://s.apache.org/hbase-2.3.1-jira To download please follow the links and instructions on our website: https://hbase.apache.org/downloads.html Questions, comments, and problems are always welcome at: dev@hbase.apache.org. Thanks to all who contributed and made this release possible. Cheers, The HBase Dev Team
[jira] [Resolved] (HBASE-24926) Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false
[ https://issues.apache.org/jira/browse/HBASE-24926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-24926. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.2+. Thanks [~vjasani] for reviewing. > Should call setFailure in MergeTableRegionsProcedure when isMergeable returns > false > --- > > Key: HBASE-24926 > URL: https://issues.apache.org/jira/browse/HBASE-24926 > Project: HBase > Issue Type: Bug > Components: master, proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.6, 2.3.2 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24686) [LOG] Log improvement in Connection#close
[ https://issues.apache.org/jira/browse/HBASE-24686?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Viraj Jasani resolved HBASE-24686. -- Fix Version/s: 2.3.2 2.2.7 2.4.0 3.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed > [LOG] Log improvement in Connection#close > - > > Key: HBASE-24686 > URL: https://issues.apache.org/jira/browse/HBASE-24686 > Project: HBase > Issue Type: Improvement > Components: Client, logging >Affects Versions: 2.2.3 >Reporter: mokai >Assignee: mokai >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.7, 2.3.2 > > > We met some customers used hbase connection improperly, some threads call > failed since the shared connection closed by one of the threads. > It's better to print the details when connection closing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24883) Migrate hbase-connectors testing to ci-hadoop
[ https://issues.apache.org/jira/browse/HBASE-24883?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-24883. --- Fix Version/s: connector-1.0.1 Resolution: Fixed Merged to master and changed repository URL on ci-hadoop. Thanks [~zhangduo] and [~busbey] for the reviews! > Migrate hbase-connectors testing to ci-hadoop > - > > Key: HBASE-24883 > URL: https://issues.apache.org/jira/browse/HBASE-24883 > Project: HBase > Issue Type: Task > Components: build, hbase-connectors >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > Fix For: connector-1.0.1 > > > Move hbase-connectors GitHub Pull Request build to ci-hadoop Jenkins master. > The new job should be based on a Jenkinsfile similarly to the main > repository. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24926) Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false
Duo Zhang created HBASE-24926: - Summary: Should call setFailure in MergeTableRegionsProcedure when isMergeable returns false Key: HBASE-24926 URL: https://issues.apache.org/jira/browse/HBASE-24926 Project: HBase Issue Type: Bug Components: master, proc-v2 Reporter: Duo Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24912) Enlarge MemstoreFlusherChore/CompactionChecker period for unit test
[ https://issues.apache.org/jira/browse/HBASE-24912?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24912. Fix Version/s: 2.3.2 2.2.6 2.4.0 3.0.0-alpha-1 Resolution: Fixed Pushed to branch-2.2+. Thanks [~stack] for reviewing. > Enlarge MemstoreFlusherChore/CompactionChecker period for unit test > --- > > Key: HBASE-24912 > URL: https://issues.apache.org/jira/browse/HBASE-24912 > Project: HBase > Issue Type: Improvement >Reporter: Guanghao Zhang >Priority: Major > Fix For: 3.0.0-alpha-1, 2.4.0, 2.2.6, 2.3.2 > > > Too many debug logs when run unit test now. > > 2020-08-19 01:20:59,899 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms. > 2020-08-19 01:20:59,899 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms. > 2020-08-19 01:20:59,900 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms. > 2020-08-19 01:20:59,900 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms. > 2020-08-19 01:20:59,905 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): MemstoreFlusherChore execution time: 0 ms. > 2020-08-19 01:20:59,905 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms. > 2020-08-19 01:21:00,001 DEBUG [regionserver/asf909:0.Chore.1] > hbase.ScheduledChore(192): CompactionChecker execution time: 0 ms. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24919) A tool to rewrite corrupted HFiles
[ https://issues.apache.org/jira/browse/HBASE-24919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anoop Sam John resolved HBASE-24919. Resolution: Duplicate Dup of HBASE-24920 > A tool to rewrite corrupted HFiles > -- > > Key: HBASE-24919 > URL: https://issues.apache.org/jira/browse/HBASE-24919 > Project: HBase > Issue Type: Brainstorming > Components: hbase-operator-tools >Reporter: Andrey Elenskiy >Priority: Major > > Typically I have been dealing with corrupted HFiles (due to loss of hdfs > blocks) by just removing them. However, It always seemed wasteful to throw > away the entire HFile (which can be hundreds of gigabytes), just because one > hdfs block is missing (128MB). > I think there's a possibility for a tool that can rewrite an HFile by > skipping corrupted blocks. > There can be multiple types of issues with hdfs blocks but any of them can be > treated as if the block doesn't exist: > 1. All the replicas can be lost > 2. The block can be corrupted due to some bug in hdfs (I've recently run into > HDFS-15186 by experimenting with EC). > At the simplest the tool can be a local mapreduce job (mapper only) with a > custom HFile reader input that can seek to next DATABLK to skip corrupted > hdfs blocks. -- This message was sent by Atlassian Jira (v8.3.4#803005)
RE: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically
Thank you sir for your very informative response. > So, a 'god' command that will fix any issue found? Yes, the idea was to make -fixInconsistencies god command, fixing all inconstancies reported in both CJ and hbck chore. But as you rightly pointed out that we are not yet confident the repair won't cause more damage than it fixes, on second thought automating fixes at this point may not be appropriate as efficacy of all the fixes is not yet proved. > Are you regularly experiencing consistency issues? If so, what version of > hbase are you running? We have experienced some inconsistencies like unknown servers, double assignment or RIT (HBASE-24885). In future we may witness some other inconsistencies, so we should make HBCK2 mature enough to handle them. Off course we will learn from the new bug and fix them in code rather than depending on HBCK2. Currently we are upgrading from 1.x to 2.2.3 version. Just a thought process to ease HBase operator job as per some operator feedback, we can discuss and come up with some ideas to handle this. Thanks -Arshad -Original Message- From: Stack [mailto:st...@duboce.net] Sent: Tuesday, August 18, 2020 9:20 AM To: Hbase-User Cc: dev@hbase.apache.org Subject: Re: Fixing Catalog Janitor and HBCK chore reported inconsistencies automatically On Thu, Aug 6, 2020 at 10:10 PM Mohammad arshad wrote: > Hello HBase Folks > > Currently Catalog Janitor (CJ) and HBCK chore reported inconsistencies > to be fixed by manually by executing HBCK2 commands. > HBCK2 requires high HBase skills. It is bit difficult for maintenance > personals to figure out which command, when and in which order to be > executed. > > True. Is there any effort going on in community to automate fixing these > inconsistencies? I also would like to contribute there. > > I was thinking, maybe we can expose CJ and HBCK chore reported > inconsistencies through a new master API and then provide option to > fix these inconsistencies. Basically adding two new commands in HBCK2 > -listInconsistencies list CJ and HBCK chore reported inconsistencies > -fixInconsistencies fix CJ and HBCK chore reported inconsistencies > (Not sure if possible to fix all inconsistencies, need to analyze all > inconsistencies case by case, but some are very straight forward for > example holes and overlap) > > So, a 'god' command that will fix any issue found? That is tough. You've seen the philosophy section on hbck2, of how it makes no claims to being so capable [1]? We are trying to get to a place where hbck2 is increasingly less necessary. The general idea is that inconsistencies are caused by bugs or oversight. As time goes by, we've been plugging the holes. Upgrading hbase gains you the fixes making the need for hbck2 less. But as you state above, when there is an issue, it can be hard for the operator to figure how to make fixes. We've been trying to improve this state with documentation in the UI up on the 'HBCK Report' page and elsewhere but there is room for improvement. We've also been trying to aggregate on the hbck2 side so that commands become increasingly 'macro', fixing a whole category of problem types rather than an affliction at a time. This should make the tool easier to use. The 'fixMeta' command is a good example here as it fixes any holes or overlaps found in hbase:meta (This is probably ripe for conversion into an auto-repair run on occasion by the Master). Another way in which we've been trying to make improvement is by obsoleting commands in hbck2 as we fix the root cause that required the hbck2 command option to be needed in the first place. CJ and the HBCK Chore can report on inconsistencies found. It is another thing altogether having them go ahead and repair any issues found mostly because we are not yet confident the repair won't cause more damage than it fixes. > Any thoughts/inputs highly appreciated. > > Are you regularly experiencing consistency issues? If so, what version of hbase are you running? Thanks, S 1. https://github.com/apache/hbase-operator-tools/tree/master/hbase-hbck2#philosophy > Regards > -Arshad > > >
