[jira] [Resolved] (HBASE-23949) refactor loadBalancer implements for rsgroup balance by table to achieve overallbalanced
[
https://issues.apache.org/jira/browse/HBASE-23949?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guanghao Zhang resolved HBASE-23949.
Fix Version/s: 2.2.5
2.4.0
2.3.0
3.0.0
Resolution: Fixed
Pushed to branch-2.2+. Thanks [~niuyulin] for contributing.
> refactor loadBalancer implements for rsgroup balance by table to achieve
> overallbalanced
> --
>
> Key: HBASE-23949
> URL: https://issues.apache.org/jira/browse/HBASE-23949
> Project: HBase
> Issue Type: Bug
> Components: rsgroup
>Affects Versions: 2.2.0
>Reporter: niuyulin
>Assignee: niuyulin
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0, 2.2.5
>
>
> now can not achieve overallbalanced when use rsgroup balancer and by table
> is on,
> because balance every table actually use the clusterload only contain one
> table's load.
> we should use clusterload contain all this rsgroup table's load to balance
> overall
>
> hbase/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java
> {code:java}
> public boolean balance(boolean force) throws IOException {
> ..
> boolean isByTable =
> getConfiguration().getBoolean("hbase.master.loadbalance.bytable", false);
> Map>> assignments =
> this.assignmentManager.getRegionStates()
> .getAssignmentsForBalancer(tableStateManager,
> this.serverManager.getOnlineServersList(),
> isByTable);
> for (Map> serverMap : assignments.values()) {
>
> serverMap.keySet().removeAll(this.serverManager.getDrainingServersList());
> }
> //Give the balancer the current cluster state.
> this.balancer.setClusterMetrics(getClusterMetricsWithoutCoprocessor());
> this.balancer.setClusterLoad(assignments);
> List plans = new ArrayList<>();
> for (Entry>> e :
> assignments.entrySet()) {
> List partialPlans =
> this.balancer.balanceCluster(e.getKey(), e.getValue());
> if (partialPlans != null) {
> plans.addAll(partialPlans);
> }
> }
> {code}
> now do refactor:
> # add method 'balanceTable' in interface LoadBalancer
> # SimpleLoadBalancer and StochasticLoadBalancer do the real 'balanceTable' ,
> and 'balanceTable' is not support in BaseLoadBalancer and
> RSGroupBasedLoadBalancer
> # RSGroupBasedLoadBalancer invoke balanceCluster , and pass GroupClusterLoad
> to internal balacer by group
> # internal balancer balance cluster invoke 'balanceTable'
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24037) Add ut for root dir and wal root dir are different
[ https://issues.apache.org/jira/browse/HBASE-24037?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guanghao Zhang resolved HBASE-24037. Fix Version/s: 2.4.0 2.3.0 3.0.0 Resolution: Fixed Pushed to branch-2.3+. > Add ut for root dir and wal root dir are different > -- > > Key: HBASE-24037 > URL: https://issues.apache.org/jira/browse/HBASE-24037 > Project: HBase > Issue Type: Sub-task >Reporter: Guanghao Zhang >Assignee: Guanghao Zhang >Priority: Major > Fix For: 3.0.0, 2.3.0, 2.4.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-24042) Branch-1 PreCommit XML check is broken
[
https://issues.apache.org/jira/browse/HBASE-24042?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-24042.
--
Resolution: Duplicate
> Branch-1 PreCommit XML check is broken
> --
>
> Key: HBASE-24042
> URL: https://issues.apache.org/jira/browse/HBASE-24042
> Project: HBase
> Issue Type: Bug
> Components: build
>Affects Versions: 1.7.0
>Reporter: Nick Dimiduk
>Priority: Major
>
> GitHub PreCommit on branch-1 appears to be missing a dependency for
> performing XML checks.
> {noformat}
> pom.xml:
> script engine for language js can not be found
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24043) [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
[
https://issues.apache.org/jira/browse/HBASE-24043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Stack resolved HBASE-24043.
---
Fix Version/s: 2.2.5
2.3.0
3.0.0
Resolution: Fixed
I just pushed this set of test changes. Want to see how it does in our flakey
list. Pushed to branch-2.2+
> [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster
> fixes and debug
> ---
>
> Key: HBASE-24043
> URL: https://issues.apache.org/jira/browse/HBASE-24043
> Project: HBase
> Issue Type: Bug
> Components: flakies
>Reporter: Michael Stack
>Assignee: Michael Stack
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.2.5
>
>
> Add some fixes, some hopeful changes that will hopeful result in fixes, and
> then a bit of debug:
> {code}
>
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
> Edit of log about archiving that shows in middle of a table create;
> try to make it less disorientating.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
> Loosen assert. Compaction may have produced a single file only. Allow
> for this.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
> Make this test less furious given it is inline w/ a bunch of unit
> tests.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
> Add debug
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
> Add wait on quota table to show up before moving forward; otherwise,
> attempt at quota setting fails.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
> Debug
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
> Remove asserts that expected regions to still have a presence in fs
> after merge when a catalogjanitor may have cleaned up parent dirs.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
> Catch exception on way out and log it rather than let it fail test.
>
> hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
> Wait on acl table before proceeding.
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24043) [Flakey Tests] TestAsyncRegionAdminApi, TestRegionMergeTransactionOnCluster fixes and debug
Michael Stack created HBASE-24043:
-
Summary: [Flakey Tests] TestAsyncRegionAdminApi,
TestRegionMergeTransactionOnCluster fixes and debug
Key: HBASE-24043
URL: https://issues.apache.org/jira/browse/HBASE-24043
Project: HBase
Issue Type: Bug
Components: flakies
Reporter: Michael Stack
Assignee: Michael Stack
Add some fixes, some hopeful changes that will hopeful result in fixes, and
then a bit of debug:
{code}
hbase-server/src/main/java/org/apache/hadoop/hbase/master/procedure/DeleteTableProcedure.java
Edit of log about archiving that shows in middle of a table create;
try to make it less disorientating.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncRegionAdminApi.java
Loosen assert. Compaction may have produced a single file only. Allow
for this.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestAsyncTableGetMultiThreaded.java
Make this test less furious given it is inline w/ a bunch of unit
tests.
hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestFromClientSide3.java
Add debug
hbase-server/src/test/java/org/apache/hadoop/hbase/quotas/TestQuotaObserverChoreRegionReports.java
Add wait on quota table to show up before moving forward; otherwise,
attempt at quota setting fails.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestHRegion.java
Debug
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionMergeTransactionOnCluster.java
Remove asserts that expected regions to still have a presence in fs
after merge when a catalogjanitor may have cleaned up parent dirs.
hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestRegionReplicas.java
Catch exception on way out and log it rather than let it fail test.
hbase-server/src/test/java/org/apache/hadoop/hbase/security/access/TestSnapshotScannerHDFSAclController.java
Wait on acl table before proceeding.
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24002) shadedjars check does not propagate --hadoop-profile
[
https://issues.apache.org/jira/browse/HBASE-24002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-24002.
--
Resolution: Fixed
> shadedjars check does not propagate --hadoop-profile
>
>
> Key: HBASE-24002
> URL: https://issues.apache.org/jira/browse/HBASE-24002
> Project: HBase
> Issue Type: Task
> Components: build
>Affects Versions: 3.0.0, 2.3.0, 2.4.0
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
> Fix For: 3.0.0, 2.3.0, 2.4.0
>
>
> After HBASE-23829, we see the {{shadedjars}} check fail on JDK11 stages. From
> the log
> {noformat}
> Tue Mar 17 00:14:24 UTC 2020
> cd
> /home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-jdk11-hadoop3-check/src
> /opt/maven/bin/mvn --batch-mode
> -Dmaven.repo.local=/home/jenkins/jenkins-slave/workspace/Base-PreCommit-GitHub-PR_PR-1296/yetus-m2/hbase-branch-2-patch-1
> clean verify -fae --batch-mode -pl
> hbase-shaded/hbase-shaded-check-invariants -am -Dtest=NoUnitTests
> -DHBasePatchProcess -Prelease -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dspotbugs.skip=true
> ...
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:enforce (hadoop3-profile-required)
> @ hbase ---
> [INFO] Adding ignore: module-info
> [WARNING] Rule 0: org.apache.maven.plugins.enforcer.RequireProperty failed
> with message:
> HBase with JDK11 requires Hadoop3. Activate the profile with
> `-Dhadoop.profile=3.0`.
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
Re: Request for Apache Hbase slack channel -- apache-hbase.slack.com
Thank you. On Tue, Mar 24, 2020, 2:47 AM Nick Dimiduk wrote: > Invitation sent. > > On Sat, Mar 21, 2020 at 11:07 PM mallik.v.ar...@gmail.com < > mallik.v.ar...@gmail.com> wrote: > > > Invite* > > --- > > Mallikarjun > > > > > > On Sun, Mar 22, 2020 at 11:36 AM mallik.v.ar...@gmail.com < > > mallik.v.ar...@gmail.com> wrote: > > > > > > > > --- > > > Mallikarjun > > > > > >
[jira] [Resolved] (HBASE-23980) Use enforcer plugin to print JVM info in maven output
[
https://issues.apache.org/jira/browse/HBASE-23980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nick Dimiduk resolved HBASE-23980.
--
Resolution: Fixed
> Use enforcer plugin to print JVM info in maven output
> -
>
> Key: HBASE-23980
> URL: https://issues.apache.org/jira/browse/HBASE-23980
> Project: HBase
> Issue Type: Task
> Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Minor
> Fix For: 3.0.0, 2.3.0, 1.7.0, 2.4.0
>
>
> This is a little nice-to-have. We currently dump OS info using
> {{os-maven-plugin}}. Since we build on multiple JVM environments, would be
> nice to add that info too. Looking around, there wasn't an obvious source of
> this information, until I realized it's readily available via
> {{enforcer:display-info}}. Should add this to the top-level pom, if we can
> get it to run just once/build instead of with every module.
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @
> hbase-archetype-builder ---
> [INFO] Maven Version: 3.6.3
> [INFO] JDK Version: 1.8.0_222 normalized as: 1.8.0-222
> [INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
> [INFO]
>
> {noformat}
> {noformat}
> [INFO] --- maven-enforcer-plugin:3.0.0-M2:display-info (default-cli) @
> hbase-archetype-builder ---
> [INFO] Maven Version: 3.6.3
> [INFO] JDK Version: 11.0.4 normalized as: 11.0.4
> [INFO] OS Info: Arch: x86_64 Family: mac Name: mac os x Version: 10.14.6
> [INFO]
>
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24042) Branch-1 PreCommit XML check is broken
Nick Dimiduk created HBASE-24042:
Summary: Branch-1 PreCommit XML check is broken
Key: HBASE-24042
URL: https://issues.apache.org/jira/browse/HBASE-24042
Project: HBase
Issue Type: Bug
Components: build
Affects Versions: 1.6.0
Reporter: Nick Dimiduk
GitHub PreCommit on branch-1 appears to be missing a dependency for performing
XML checks.
{noformat}
pom.xml:
script engine for language js can not be found
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24041) [regression] Increase RESTServer buffer size back to 64k
Esteban Gutierrez created HBASE-24041:
-
Summary: [regression] Increase RESTServer buffer size back to 64k
Key: HBASE-24041
URL: https://issues.apache.org/jira/browse/HBASE-24041
Project: HBase
Issue Type: Bug
Affects Versions: 2.2.0, 3.0.0, 2.3.0, 2.4.0
Reporter: Esteban Gutierrez
HBASE-14492 is not longer present in our current releases after HBASE-12894.
Unfortunately our RESTServer is not extending HttpServer which means that
{{DEFAULT_MAX_HEADER_SIZE}} is not being set and HTTP requests with a very
large header can still cause connection issues for clients. A quick fix is just
to add the settings to the {{HttpConfiguration}} configuration object. A long
term solution should be to re-factor services that create an HTTP server and
normalize all configuration settings across all of them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24032) [RSGroup] Assign created tables to respective rsgroup automatically instead of manual operations
[ https://issues.apache.org/jira/browse/HBASE-24032?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-24032. --- Hadoop Flags: Reviewed Resolution: Fixed > [RSGroup] Assign created tables to respective rsgroup automatically instead > of manual operations > - > > Key: HBASE-24032 > URL: https://issues.apache.org/jira/browse/HBASE-24032 > Project: HBase > Issue Type: Improvement > Components: master, rsgroup >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Major > Fix For: 2.3.0, master, 1.7.0, 2.2.4, 1.4.14 > > > In prod practise, it's time consuming and tiring to assign tables to their > rsgroups for HBase admin by shell, especially in 1.x rsgroup client is > IA.private, we could not even write some programs to do that. > And I'm here proposing a solution to do the automated rsgroup assignment by > allowing running a shell in postCreateTable to determine which table should > go to which rsgroup (way in branch-1 not master). The benefit is obvious, > admin can determine or change at any time the mapping rule in script without > restarting or re-config anything. It's flexible, convenience and time saving. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24040) WALFactory.Providers.multiwal causes StackOverflowError
wenfeiyi666 created HBASE-24040:
---
Summary: WALFactory.Providers.multiwal causes StackOverflowError
Key: HBASE-24040
URL: https://issues.apache.org/jira/browse/HBASE-24040
Project: HBase
Issue Type: Bug
Components: wal
Affects Versions: 2.0.0, 1.0.0
Reporter: wenfeiyi666
Assignee: wenfeiyi666
Fix For: 2.0.0, 1.0.0
if 'hbase.wal.provider=multiwal' and
'hbase.wal.regiongrouping.delegate.provider=multiwal', regionserver will create
RegionGroupingProvider in a loop until StackOverflowError.
{code:java}
2020-03-24 19:08:19,180 WARN [RS_CLOSE_META-regionserver/localhost:16020-0]
handler.AssignRegionHandler: Fatal error occurred while opening region
hbase:meta,,1.1
588230740, aborting...
java.lang.StackOverflowError
at
java.lang.AbstractStringBuilder.ensureCapacityInternal(AbstractStringBuilder.java:125)
at
java.lang.AbstractStringBuilder.append(AbstractStringBuilder.java:566)
at java.lang.StringBuffer.append(StringBuffer.java:357)
at
org.apache.log4j.helpers.ISO8601DateFormat.format(ISO8601DateFormat.java:134)
at java.text.DateFormat.format(DateFormat.java:345)
at
org.apache.log4j.helpers.PatternParser$DatePatternConverter.convert(PatternParser.java:443)
at
org.apache.log4j.helpers.PatternConverter.format(PatternConverter.java:65)
at org.apache.log4j.PatternLayout.format(PatternLayout.java:506)
at org.apache.log4j.WriterAppender.subAppend(WriterAppender.java:310)
at
org.apache.log4j.RollingFileAppender.subAppend(RollingFileAppender.java:276)
at org.apache.log4j.WriterAppender.append(WriterAppender.java:162)
at org.apache.log4j.AppenderSkeleton.doAppend(AppenderSkeleton.java:251)
at
org.apache.log4j.helpers.AppenderAttachableImpl.appendLoopOnAppenders(AppenderAttachableImpl.java:66)
at org.apache.log4j.Category.callAppenders(Category.java:206)
at org.apache.log4j.Category.forcedLog(Category.java:391)
at org.apache.log4j.Category.log(Category.java:856)
at org.slf4j.impl.Log4jLoggerAdapter.info(Log4jLoggerAdapter.java:304)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getStrategy(RegionGroupingProvider.java:105)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.init(RegionGroupingProvider.java:160)
at
org.apache.hadoop.hbase.wal.WALFactory.createProvider(WALFactory.java:161)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.createProvider(RegionGroupingProvider.java:166)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:184)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:212)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:192)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:212)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:192)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:212)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:192)
at
org.apache.hadoop.hbase.wal.RegionGroupingProvider.getWAL(RegionGroupingProvider.java:212)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24039) HBCK2 feature negotiation to check what commands are supported
Richard Antal created HBASE-24039: - Summary: HBCK2 feature negotiation to check what commands are supported Key: HBASE-24039 URL: https://issues.apache.org/jira/browse/HBASE-24039 Project: HBase Issue Type: Improvement Reporter: Richard Antal Each version of HBase is shipped with a specific version of HBCK2 therefore we can check the version and compare it to the minimum version for the given command. It is not that easy to decide if the command is supported or not when dealing with back-ported features. It the conclusion of this I would recommend using feature negotiation to check what commands are supported instead of comparing versions. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-22740) [RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x
[ https://issues.apache.org/jira/browse/HBASE-22740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Reid Chan resolved HBASE-22740. --- Resolution: Fixed > [RSGroup] Forward-port HBASE-22658 to master branch and branch-2.x > -- > > Key: HBASE-22740 > URL: https://issues.apache.org/jira/browse/HBASE-22740 > Project: HBase > Issue Type: Bug > Components: rsgroup >Affects Versions: 2.0.6, 2.2.3, 2.1.9 >Reporter: Reid Chan >Assignee: Reid Chan >Priority: Major > Fix For: master > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-24038) Add a metric to show the locality of ssd in table.jsp
Zheng Wang created HBASE-24038: -- Summary: Add a metric to show the locality of ssd in table.jsp Key: HBASE-24038 URL: https://issues.apache.org/jira/browse/HBASE-24038 Project: HBase Issue Type: Improvement Components: monitoring Reporter: Zheng Wang Assignee: Zheng Wang If a hdfs block has a replica on local hdd and a replica on remote ssd, then the local one is first priority, so i think it is useful to show the locality of ssd, especially when use the ONE_SSD storagy policy. I have developed a patch, will submit it later. -- This message was sent by Atlassian Jira (v8.3.4#803005)
