Re: [VOTE] The first HBase 2.2.4 release candidate (RC0) is available
+1 from me. With 3 binding +1s and 1 non-binding +1, the vote passes. Let me push out the release. Jan Hentschel 于2020年3月13日周五 下午10:22写道： > +1 (binding) > > * Signature: ok > * Checksum : ok > * Rat check (1.8.0_202-ea): ok > - mvn clean apache-rat:check > * Built from source (1.8.0_202-ea): ok > - mvn clean install -DskipTests > * Unit tests pass (1.8.0_202-ea): ok > - mvn package -P runSmallTests > > From: Guanghao Zhang > Reply-To: "dev@hbase.apache.org" > Date: Wednesday, March 11, 2020 at 8:03 AM > To: HBase Dev List , Hbase-User < > u...@hbase.apache.org> > Subject: [VOTE] The first HBase 2.2.4 release candidate (RC0) is available > > Please vote on this release candidate (RC) for Apache HBase 2.2.4. > > The VOTE will remain open for at least 72 hours. > [ ] +1 Release this package as Apache HBase 2.2.4 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 2.2.4RC0. The release files, including > signatures, digests, etc. can be found at: > https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/ > > Maven artifacts are available in a staging repository at: > https://repository.apache.org/content/repositories/orgapachehbase-1385/ > > Signatures used for HBase RCs can be found in this file: > https://dist.apache.org/repos/dist/release/hbase/KEYS > > The list of bug fixes going into 2.2.4 can be found in included > CHANGES.md and RELEASENOTES.md available here: > https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/CHANGES.md > https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/RELEASENOTES.md > > A detailed source and binary compatibility report for this release is > available at: > > https://dist.apache.org/repos/dist/dev/hbase/2.2.4RC0/api_compare_2.2.4RC0_to_2.2.3.html > > To learn more about Apache HBase, please see http://hbase.apache.org/ > > > Thanks, > Guanghao Zhang > >
[jira] [Created] (HBASE-24023) Add 2.2.4 to download page
Guanghao Zhang created HBASE-24023: -- Summary: Add 2.2.4 to download page Key: HBASE-24023 URL: https://issues.apache.org/jira/browse/HBASE-24023 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24022) Set version as 2.2.5-SNAPSHOT in branch-2.2
Guanghao Zhang created HBASE-24022: -- Summary: Set version as 2.2.5-SNAPSHOT in branch-2.2 Key: HBASE-24022 URL: https://issues.apache.org/jira/browse/HBASE-24022 Project: HBase Issue Type: Sub-task Reporter: Guanghao Zhang -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24022) Set version as 2.2.5-SNAPSHOT in branch-2.2
[ https://issues.apache.org/jira/browse/HBASE-24022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24022. Fix Version/s: 2.2.5 Assignee: Guanghao Zhang Resolution: Fixed > Set version as 2.2.5-SNAPSHOT in branch-2.2 > --- > > Key: HBASE-24022 > URL: https://issues.apache.org/jira/browse/HBASE-24022 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.5 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24024) Optionally reject request with very high no of rows
Viraj Jasani created HBASE-24024: Summary: Optionally reject request with very high no of rows Key: HBASE-24024 URL: https://issues.apache.org/jira/browse/HBASE-24024 Project: HBase Issue Type: Improvement Reporter: Viraj Jasani Assignee: Viraj Jasani As part of multi() request, RegionServer might receive requests with higher rows than row size threshold defined by hbase.rpc.rows.warning.threshold, and we provide warning log. To improve CPU consumption, we should introduce a config that would let RegionServer reject such requests other than providing warning log. This is an attempt to not let any client overburden CPU computation available to RegionServers. -- This message was sent by Atlassian Jira (v8.3.4#803005)
BulkDeleteEndpoint not deleting rows
I am using bulkdelete endpoint coprocessor . After running , in logs it is printing number of rows deleted but when I go to hbase shell and scan table , all rows are present. -- Sent from: http://apache-hbase.679495.n3.nabble.com/HBase-Developer-f679493.html
[jira] [Created] (HBASE-24025) Improve performance of move_servers_rsgroup and move_tables_rsgroup by using async region move API
Mohammad Arshad created HBASE-24025: --- Summary: Improve performance of move_servers_rsgroup and move_tables_rsgroup by using async region move API Key: HBASE-24025 URL: https://issues.apache.org/jira/browse/HBASE-24025 Project: HBase Issue Type: Improvement Components: rsgroup Reporter: Mohammad Arshad Assignee: Mohammad Arshad Currently move_servers_rsgroup and move_tables_rsgroup commands and APIs are taking lot of time. In my test environment, to move a server with 100 regions it takes around 137 seconds. Similarly it takes around same time to move a table with 100 regions to other group. The time taken in rsgroup meta update is negligible. Almost all the time is taken in region moment. This is happening because region is moved serially using getAssignmentManager().move(region) API API getAssignmentManager().moveAsync(regionplan) can be used to move the regions in parallel to improve the performance of region group move servers and tables commands and APIs -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24023) Add 2.2.4 to download page
[ https://issues.apache.org/jira/browse/HBASE-24023?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24023. Resolution: Fixed > Add 2.2.4 to download page > -- > > Key: HBASE-24023 > URL: https://issues.apache.org/jira/browse/HBASE-24023 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-23922) Release 2.2.4
[ https://issues.apache.org/jira/browse/HBASE-23922?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-23922. Fix Version/s: 2.2.4 Resolution: Fixed > Release 2.2.4 > - > > Key: HBASE-23922 > URL: https://issues.apache.org/jira/browse/HBASE-23922 > Project: HBase > Issue Type: Umbrella >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 2.2.4 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24006) Get `-PrunMediumTests` passing on JDK11
[ https://issues.apache.org/jira/browse/HBASE-24006?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nick Dimiduk resolved HBASE-24006. -- Assignee: Nick Dimiduk Resolution: Not A Problem Works for me on top of HBASE-23829 > Get `-PrunMediumTests` passing on JDK11 > --- > > Key: HBASE-24006 > URL: https://issues.apache.org/jira/browse/HBASE-24006 > Project: HBase > Issue Type: Sub-task > Components: test >Affects Versions: 3.0.0, 2.3.0 >Reporter: Nick Dimiduk >Assignee: Nick Dimiduk >Priority: Major > > Build on HBASE-23829, now looking at medium tests. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24026) Revisit cluster shutdown
Michael Stack created HBASE-24026: - Summary: Revisit cluster shutdown Key: HBASE-24026 URL: https://issues.apache.org/jira/browse/HBASE-24026 Project: HBase Issue Type: Improvement Components: shutdown Reporter: Michael Stack On review of HBASE-23984, [~zhangduo] suggested review/redo of cluster shutdown altogether; a welcome but probably substantial undertaking. Spec/suggestion to follow. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24027) Spotbugs: Return value of putIfAbsent is ignored
Wei-Chiu Chuang created HBASE-24027:
---
Summary: Spotbugs: Return value of putIfAbsent is ignored
Key: HBASE-24027
URL: https://issues.apache.org/jira/browse/HBASE-24027
Project: HBase
Issue Type: Bug
Components: master
Reporter: Wei-Chiu Chuang
Looks like a regression from HBASE-23561.
[https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1301/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html]
{quote}
Return value of putIfAbsent is ignored, but node is reused in
org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
Bug type RV_RETURN_VALUE_OF_PUTIFABSENT_IGNORED (click for details)
In class org.apache.hadoop.hbase.master.assignment.RegionStates
In method
org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
Called method java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(Object,
Object)
Type org.apache.hadoop.hbase.master.assignment.RegionStateNode
Value loaded from node
At RegionStates.java:[line 133]
{quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24027) Spotbugs: Return value of putIfAbsent is ignored
[
https://issues.apache.org/jira/browse/HBASE-24027?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jan Hentschel resolved HBASE-24027.
---
Resolution: Duplicate
This one duplicates HBASE-23983.
> Spotbugs: Return value of putIfAbsent is ignored
>
>
> Key: HBASE-24027
> URL: https://issues.apache.org/jira/browse/HBASE-24027
> Project: HBase
> Issue Type: Bug
> Components: master
>Reporter: Wei-Chiu Chuang
>Priority: Major
>
> Looks like a regression from HBASE-23561.
> [https://builds.apache.org/job/HBase-PreCommit-GitHub-PR/job/PR-1301/2/artifact/yetus-general-check/output/branch-spotbugs-hbase-server-warnings.html]
> {quote}
> Return value of putIfAbsent is ignored, but node is reused in
> org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
> Bug type RV_RETURN_VALUE_OF_PUTIFABSENT_IGNORED (click for details)
> In class org.apache.hadoop.hbase.master.assignment.RegionStates
> In method
> org.apache.hadoop.hbase.master.assignment.RegionStates.createRegionStateNode(RegionInfo)
> Called method java.util.concurrent.ConcurrentSkipListMap.putIfAbsent(Object,
> Object)
> Type org.apache.hadoop.hbase.master.assignment.RegionStateNode
> Value loaded from node
> At RegionStates.java:[line 133]
> {quote}
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-23984) [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown
[
https://issues.apache.org/jira/browse/HBASE-23984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-23984.
---
Fix Version/s: 2.3.0
3.0.0
Hadoop Flags: Reviewed
Assignee: Michael Stack
Resolution: Fixed
Pushed on branch-2.3, branch-2, and master (didn't mark this 2.4.0 as per
comment by Duo up on dev list... until there is a 2.3.0 and an issue is in
both, then its 2.3.0, not 2.4.0).
Thanks for the reviews. Linked follow-ons to this issue.
> [Flakey Tests] TestMasterAbortAndRSGotKilled fails in teardown
> --
>
> Key: HBASE-23984
> URL: https://issues.apache.org/jira/browse/HBASE-23984
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.3.0, 2.4.0
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0
>
> Attachments:
> 0001-HBASE-23984-Flakey-Tests-TestMasterAbortAndRSGotKill.patch, Screen Shot
> 2020-03-17 at 9.46.49 PM.png
>
>
> Its failing with decent frequency of late in shutdown of cluster. Seems
> basic. There is an unassign/move going on. Test just checks Master can come
> back up after being killed. Does not check move is done. If on subsequent
> cluster shutdown, if the move can't report the Master because its shutting
> down, then the move fails, we abort the server, and then we get a wonky loop
> where we can't close because server is aborting.
> At the root, there is a misaccounting when the unassign close fails where we
> don't cleanup references in the regionserver local RIT accounting. Deeper
> than this, close code is duplicated in three places that I can see; in
> RegionServer, in CloseRegionHandler, and in UnassignRegionHandler.
> Let me fix this issue and the code dupe.
> Details:
> From
> https://builds.apache.org/job/HBase-Flaky-Tests/job/branch-2/5733/artifact/hbase-server/target/surefire-reports/org.apache.hadoop.hbase.master.TestMasterAbortAndRSGotKilled-output.txt
> Here is the unassign handler failing because master went down earlier (Its
> probably trying to talk to the old Master location)
> {code}
> * ABORTING region server asf905.gq1.ygridcore.net,32989,1584000644108:
> Failed to close region ede67f9f661acc1241faf468b081d548 and can not recover
> *
> Cause:
> java.io.IOException: Failed to report close to master:
> ede67f9f661acc1241faf468b081d548
> at
> org.apache.hadoop.hbase.regionserver.handler.UnassignRegionHandler.process(UnassignRegionHandler.java:125)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
> ... then the cluster shutdown tries to close the same Region... but fails
> because we are aborting because of above
> {code}
> 2020-03-12 08:11:16,600 ERROR [RS_CLOSE_REGION-regionserver/asf905:0-0]
> helpers.MarkerIgnoringBase(159): * ABORTING region server
> asf905.gq1.ygridcore.net,32989,1584000644108: Unrecoverable exception while
> closing region
> hbase:namespace,,1584000652744.78f4ae5beda711a9bebad0b6b8376cc9., still
> finishing close *
> java.io.IOException: Aborting flush because server is aborted...
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalPrepareFlushCache(HRegion.java:2545)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2530)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2504)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2495)
> at
> org.apache.hadoop.hbase.regionserver.HRegion.doClose(HRegion.java:1650)
> at org.apache.hadoop.hbase.regionserver.HRegion.close(HRegion.java:1552)
> at
> org.apache.hadoop.hbase.regionserver.handler.CloseRegionHandler.process(CloseRegionHandler.java:110)
> at
> org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:104)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
>
> And the RS keeps looping trying to close the Region even though we're aborted
> and there is handling in RS close Regions to deal with abort.
> Trouble seems to be because when UnassignRegionHandler fails its region
> close, it does not unregister the Region with
> rs.getRegionsInTransitionInRS().remove(encodedNameBytes, Boolean.FALSE);
--
This message was sent by Atlassian Jira
(v8.3.4#80
[jira] [Created] (HBASE-24028) MapReduce on snapshot restores and opens all regions in each mapper
Xu Cang created HBASE-24028: --- Summary: MapReduce on snapshot restores and opens all regions in each mapper Key: HBASE-24028 URL: https://issues.apache.org/jira/browse/HBASE-24028 Project: HBase Issue Type: Bug Affects Versions: 1.6.0, 2.3.0 Reporter: Xu Cang Given this scenario: one MR job scans a table (with many regions). I will use 'RestoreSnapshotHelper' to restore snapshot for all regions in each mapper. In the code [https://github.com/apache/hbase/blob/branch-2.0/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/RestoreSnapshotHelper.java#L183] Seems there is no way to only restore relevant regions from snapshot to region. This leads to extreme slowness and waste of resource. Please correct me if I am wrong or miss anything. thanks. One quick example I san show as below, in my test, there are 2 regions in a testing table. and each mapper opens and iterates 2 regions. 2020-03-19 18:58:15,225 INFO [main] mapred.MapTask - Map output collector class = org.apache.hadoop.mapred.MapTask$MapOutputBuffer 2020-03-19 18:58:15,285 INFO [main] snapshot.RestoreSnapshotHelper - region to add: *d7f85b4a9d3fa22a5e7b88bda39f6d50* 2020-03-19 18:58:15,285 INFO [main] snapshot.RestoreSnapshotHelper - region to add: *69dd3fdba3698f827f8883ed911161ef* 2020-03-19 18:58:15,286 INFO [main] snapshot.RestoreSnapshotHelper - clone region=d7f85b4a9d3fa22a5e7b88bda39f6d50 as d7f85b4a9d3fa22a5e7b88bda39f6d50 So if I misunderstood anything, can anyone point to me where in this class, can distinguish which region to go through for different mappers? btw the original implementation for MR on Snapshot is here, there weren't too many big changes after that HBASE-8369 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24029) Convert logging code to use Parameters when possible
Huaxiang Sun created HBASE-24029:
Summary: Convert logging code to use Parameters when possible
Key: HBASE-24029
URL: https://issues.apache.org/jira/browse/HBASE-24029
Project: HBase
Issue Type: Improvement
Affects Versions: master, 2.4.0
Reporter: Huaxiang Sun
Based on [~ndimiduk]'s comments in HBASE-23957 ,
{code:java}
nit: we make a habit of using the logger's format string capability. That way
it only performs the computation necessary to build the log message if the
logger is emitting messages at the target level.
LOG.debug("Starting the test {}", name.getMethodName()){code}
and
[http://blog.logscape.com/2017/03/concatenation-or-parameters-both-whats-the-top-method-of-java-logging/],
I did a quick search of the code base and found
[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/snapshot/SnapshotManager.java#L334]
This needs to be converted to use parameter instead of string concatenation.
.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
