[jira] [Created] (HBASE-25849) Backport "HBASE-22738 Fallback to default group to choose RS when there are no RS in current group" and "HBASE-24760 Add a config hbase.rsgroup.fallback.enable for RSGro

[Caroline Zhou (Jira)]

Caroline Zhou created HBASE-25849:
-

 Summary: Backport "HBASE-22738 Fallback to default group to choose 
RS when there are no RS in current group" and "HBASE-24760 Add a config 
hbase.rsgroup.fallback.enable for RSGroup fallback feature" to branch-1
 Key: HBASE-25849
 URL: https://issues.apache.org/jira/browse/HBASE-25849
 Project: HBase
  Issue Type: Improvement
Reporter: Caroline Zhou
Assignee: Caroline Zhou






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25848) Add flexibility to backup replication in case replication filter throws an exception

[Sandeep Pal (Jira)]

Sandeep Pal created HBASE-25848:
---

 Summary: Add flexibility to backup replication in case replication 
filter throws an exception
 Key: HBASE-25848
 URL: https://issues.apache.org/jira/browse/HBASE-25848
 Project: HBase
  Issue Type: Bug
Affects Versions: 1.6.0, 3.0.0-alpha-1, 2.6.7
Reporter: Sandeep Pal
Assignee: Sandeep Pal


There may be situations when the wal entry stream might result in some 
temporary issues but expected to recover at some point of time. In this case, 
we should have an option to backup replication and retry until the wal entry 
filter recover instead of just aborting the region server.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-25846) RSGroupBasedLoadBalancer does not allow dynamic config setting

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang reopened HBASE-25846:
---

> RSGroupBasedLoadBalancer does not allow dynamic config setting
> --
>
> Key: HBASE-25846
> URL: https://issues.apache.org/jira/browse/HBASE-25846
> Project: HBase
>  Issue Type: Improvement
>Reporter: Caroline Zhou
>Assignee: Caroline Zhou
>Priority: Minor
>
> In branch-1, 
> [RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452]
>  doesn't do anything – it should call the internal balancer's 
> onConfigurationChange().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-25847) More DEBUG and TRACE level logging in CatalogJanitor and HbckChore

[Andrew Kyle Purtell (Jira)]

Andrew Kyle Purtell created HBASE-25847:
---

 Summary: More DEBUG and TRACE level logging in CatalogJanitor and 
HbckChore
 Key: HBASE-25847
 URL: https://issues.apache.org/jira/browse/HBASE-25847
 Project: HBase
  Issue Type: Sub-task
Affects Versions: 2.4.3
Reporter: Andrew Kyle Purtell
Assignee: Andrew Kyle Purtell
 Fix For: 3.0.0-alpha-1, 2.5.0, 2.4.3


Add DEBUG and TRACE level logging to CatalogJanitor and HbckChore to facilitate 
debugging and verification of correct operation.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-25846) RSGroupBasedLoadBalancer does not allow dynamic config setting

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-25846.
---
Resolution: Duplicate

Thanks for reporting this [~caroliney14].

It has already been fixed by HBASE-25825.

> RSGroupBasedLoadBalancer does not allow dynamic config setting
> --
>
> Key: HBASE-25846
> URL: https://issues.apache.org/jira/browse/HBASE-25846
> Project: HBase
>  Issue Type: Improvement
>Reporter: Caroline Zhou
>Assignee: Caroline Zhou
>Priority: Minor
>
> In branch-1, 
> [RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452]
>  doesn't do anything – it should call the internal balancer's 
> onConfigurationChange().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Multi-dimensional Range Queries Help

[Nick Dimiduk]

Hi Kevin,

Did you get an answer to your question, maybe over on hbase-user?

As it seems you're aware, HBase is built on a single index -- the rowkey.
You may be able to implement something like MySQL's composite indexing on
HBase if the algorithm can be mapped to a 1-dimensional linear index. You
would have to implement this yourself as HBase doesn't offer this out of
the box. Such an encoding would be an interesting contribution to HBase, it
might sit over next to our other data encoding "types" in
`org.apache.hadoop.hbase.types`.

As for why your filtered queries are slow, you're the best person to start
answering that question. Is your data local to the region server that's
hosting it, or do you have multiple network hops and service
serialize/deserialize steps in your hot path? Is your index optimized for
your query (sounds like maybe not, based on the first question)? Have you
seen the Profiling Servlet [0]? You can start by setting that up, isolating
the workload, and collecting some FlameGraphs to analyze.

Thanks,
Nick

[0]: https://hbase.apache.org/book.html#profiler

On Mon, Apr 12, 2021 at 10:26 AM Kevin Wright 
wrote:

> Hi!
>
> Our application requires fast read queries that specify two ranges. One
> range on timestamps, and another on ids. We are currently using Apache
> HBase as our db, but we’re unsure how to optimally design the row keys /
> schemas. Currently, scanning over row key (the ids) with filter on
> timeranges is taking more time than what we expect. A normal query would
> probably have say 200 rows that match the id range, and about 10 rows that
> match both ranges, and we have currently on the order of 10s of millions of
> rows.
>
> We’re wondering if there’s something we can do to increase throughput with
> HBase (e.g., is there something like composite indexing like in MySQL?).
> Not sure if this is the best place to ask this, but if anyone could point
> us to the right direction, that would be great!
>
> Thank you!
>

Re: [DISCUSS] Next 2.x releases

[Sean Busbey]

My understanding is that backup work is not ready for inclusion in 2.x.

The talk of removing it from the master branch and proposed adoption of the
feature through more involvement from some community members were not so
long ago.


On Tue, May 4, 2021, 15:49 Andrew Purtell  wrote:

> Correct me if I am mistaken but backup tests failing on master in precommit
> is common enough to warrant ignoring them as unrelated. Is it not fully
> baked yet?
>
> +1 for backport of tracing. If we do the backport to branch-2 that would be
> one new compelling reason for a 2.5.0 release.
>
>
> On Tue, May 4, 2021 at 9:34 AM Nick Dimiduk  wrote:
>
> > On Fri, Apr 30, 2021 at 5:40 PM 张铎(Duo Zhang) 
> > wrote:
> >
> > > Does anyone have interest in backporting HBASE-22120 to branch-2?
> > >
> >
> > Yes, I think there would be interest in getting your tracing effort onto
> > branch-2 ; there's quite a few "watchers" on that Jira.
> >
> > What about the backup work? Has it stabilized enough for backport?
> >
> > Conversely, if we don't have a killer new feature for 2.5, does that mean
> > it's time for 3.0?
> >
> > Andrew Purtell  于2021年5月1日周六 上午5:46写道：
> > >
> > > > Inline
> > > >
> > > > On Fri, Apr 30, 2021 at 2:11 PM Nick Dimiduk 
> > > wrote:
> > > >
> > > > > Heya,
> > > > >
> > > > > I'd like to have a planning discussion around our 2.x releases. 2.4
> > > seems
> > > > > to be humming along nicely, I think we won't have a need for 2.3
> much
> > > > > longer. Likewise, it seems like it should be time to start planning
> > > 2.5,
> > > > > but when I look at the issues unique to that branch [0], I don't
> see
> > > the
> > > > > big new feature that justifies the new minor release. Rather, I
> see a
> > > > > number of items that should be backported to a 2.4.x. Do we have a
> > big
> > > > new
> > > > > feature in the works? Should we consider backporting something from
> > > > master?
> > > > > Or maybe there's enough minor changes on 2.5 to justify the
> > release...
> > > > but
> > > > > if so, which of them motivate users to upgrade?
> > > > >
> > > > > So, to summarize:
> > > > >
> > > > > - How much longer do we want to run 2.3?
> > > > > - What issues in the below query can be backported to 2.4? Any
> > > > volunteers?
> > > > >
> > > >
> > > > Thanks for starting this discussion, Nick.
> > > >
> > > > Looking at that report, issues like HBASE-24305, HBASE-25793, or
> > > > HBASE-25458 that clean up or reduce interfaces or refactor/move
> classes
> > > are
> > > > probably excluded from a patch release by our guidelines. Conversely,
> > > they
> > > > would not provide much value if backported.
> > > >
> > > > I also agree that the motivation for 2.5 here is thin as of now.
> > > Refactors,
> > > > interface improvements, or deprecation removals will be nice-to-haves
> > > when
> > > > there is a 2.5 someday.
> > > >
> > > > All the others in the report are either operational improvements that
> > > would
> > > > be nice to backport or bug fixes that should be backported.
> > > >
> > > > There might be case by case issues with compatibility during the pick
> > > > backs, but we can deal with them one at a time.
> > > >
> > > > If you are looking for a volunteer to do this, as 2.4 RM I am game.
> > > >
> > > >
> > > > - What's the big new feature that motivates 2.5?
> > > > >
> > > > > Thanks,
> > > > > Nick
> > > > >
> > > > > [0]:
> > > > >
> > > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.5.0%20AND%20fixVersion%20not%20in%20(2.4.0%2C%202.4.1%2C%202.4.2%2C%202.4.3)%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Words like orphans lost among the crosstalk, meaning torn from
> truth's
> > > > decrepit hands
> > > >- A23, Crosstalk
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>

Re: 2.4.3 RC next week

[Andrew Purtell]

I'm still working on testing. It is probably fine, but I want to be sure.
Apologies for the delay. The artifacts are available so if you think we
should not be waiting at this point, a VOTE is possible.


On Fri, Apr 30, 2021 at 2:50 PM Andrew Purtell  wrote:

> This has been held up by HBASE-25829 and a couple of unit tests nits.
> Assessing the severity, it all seems minor, and so unless that changes I
> will probably announce the RC artifacts by end of today. They have deployed
> to the usual staging location and temporary repository, pending preflight
> checks.
>
> On Mon, Apr 19, 2021 at 4:38 PM Andrew Purtell 
> wrote:
>
>> It's about time for 2.4.3.
>>
>> There will be a RC for 2.4.3 available for voting next week. If you have
>> any pending work please let me know if you need assistance getting it
>> committed.
>>
>> --
>> Best regards,
>> Andrew
>>
>> Words like orphans lost among the crosstalk, meaning torn from truth's
>> decrepit hands
>>- A23, Crosstalk
>>
>
>
> --
> Best regards,
> Andrew
>
> Words like orphans lost among the crosstalk, meaning torn from truth's
> decrepit hands
>- A23, Crosstalk
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

Re: [DISCUSS] Next 2.x releases

[Andrew Purtell]

Correct me if I am mistaken but backup tests failing on master in precommit
is common enough to warrant ignoring them as unrelated. Is it not fully
baked yet?

+1 for backport of tracing. If we do the backport to branch-2 that would be
one new compelling reason for a 2.5.0 release.


On Tue, May 4, 2021 at 9:34 AM Nick Dimiduk  wrote:

> On Fri, Apr 30, 2021 at 5:40 PM 张铎(Duo Zhang) 
> wrote:
>
> > Does anyone have interest in backporting HBASE-22120 to branch-2?
> >
>
> Yes, I think there would be interest in getting your tracing effort onto
> branch-2 ; there's quite a few "watchers" on that Jira.
>
> What about the backup work? Has it stabilized enough for backport?
>
> Conversely, if we don't have a killer new feature for 2.5, does that mean
> it's time for 3.0?
>
> Andrew Purtell  于2021年5月1日周六 上午5:46写道：
> >
> > > Inline
> > >
> > > On Fri, Apr 30, 2021 at 2:11 PM Nick Dimiduk 
> > wrote:
> > >
> > > > Heya,
> > > >
> > > > I'd like to have a planning discussion around our 2.x releases. 2.4
> > seems
> > > > to be humming along nicely, I think we won't have a need for 2.3 much
> > > > longer. Likewise, it seems like it should be time to start planning
> > 2.5,
> > > > but when I look at the issues unique to that branch [0], I don't see
> > the
> > > > big new feature that justifies the new minor release. Rather, I see a
> > > > number of items that should be backported to a 2.4.x. Do we have a
> big
> > > new
> > > > feature in the works? Should we consider backporting something from
> > > master?
> > > > Or maybe there's enough minor changes on 2.5 to justify the
> release...
> > > but
> > > > if so, which of them motivate users to upgrade?
> > > >
> > > > So, to summarize:
> > > >
> > > > - How much longer do we want to run 2.3?
> > > > - What issues in the below query can be backported to 2.4? Any
> > > volunteers?
> > > >
> > >
> > > Thanks for starting this discussion, Nick.
> > >
> > > Looking at that report, issues like HBASE-24305, HBASE-25793, or
> > > HBASE-25458 that clean up or reduce interfaces or refactor/move classes
> > are
> > > probably excluded from a patch release by our guidelines. Conversely,
> > they
> > > would not provide much value if backported.
> > >
> > > I also agree that the motivation for 2.5 here is thin as of now.
> > Refactors,
> > > interface improvements, or deprecation removals will be nice-to-haves
> > when
> > > there is a 2.5 someday.
> > >
> > > All the others in the report are either operational improvements that
> > would
> > > be nice to backport or bug fixes that should be backported.
> > >
> > > There might be case by case issues with compatibility during the pick
> > > backs, but we can deal with them one at a time.
> > >
> > > If you are looking for a volunteer to do this, as 2.4 RM I am game.
> > >
> > >
> > > - What's the big new feature that motivates 2.5?
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > > > [0]:
> > > >
> > > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.5.0%20AND%20fixVersion%20not%20in%20(2.4.0%2C%202.4.1%2C%202.4.2%2C%202.4.3)%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Words like orphans lost among the crosstalk, meaning torn from truth's
> > > decrepit hands
> > >- A23, Crosstalk
> > >
> >
>


-- 
Best regards,
Andrew

Words like orphans lost among the crosstalk, meaning torn from truth's
decrepit hands
   - A23, Crosstalk

[jira] [Created] (HBASE-25846) RSGroupBasedLoadBalancer does not allow dynamic config setting

[Caroline Zhou (Jira)]

Caroline Zhou created HBASE-25846:
-

 Summary: RSGroupBasedLoadBalancer does not allow dynamic config 
setting
 Key: HBASE-25846
 URL: https://issues.apache.org/jira/browse/HBASE-25846
 Project: HBase
  Issue Type: Improvement
Reporter: Caroline Zhou
Assignee: Caroline Zhou


In master/branch-2, 
[RSGroupBasedLoadBalancer#onConfigurationChange|[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L411]]
 only accounts for changes in the fallback setting, and in branch-1, 
[RSGroupBasedLoadBalancer#onConfigurationChange|https://github.com/apache/hbase/blob/branch-1/hbase-rsgroup/src/main/java/org/apache/hadoop/hbase/rsgroup/RSGroupBasedLoadBalancer.java#L452]
 doesn't do anything at all. In both cases, we should also call the internal 
balancer's onConfigurationChange().



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [DISCUSS] User experience on Stripe Compaction

[Huaxiang Sun]

It is great to hear successful stories from large scale production
deployment!

On Mon, May 3, 2021 at 1:31 PM clara xiong  wrote:

> It is great to hear this is still in use  in a large production and getting
> improvement!
>
> On Tue, Apr 27, 2021 at 3:01 AM 张铎(Duo Zhang) 
> wrote:
>
> > Stripe compaction is a feature which has been implemented long ago in
> HBase
> > but I've never seen extensive usage in the community. And recently I
> found
> > that a big company in China, Meituan, has made use of stripe compaction
> in
> > their production cluster. One of the team member shared some information
> on
> > a github PR and she agreed that I could share it in the mailing list.
> >
> > Hi, @Apache9  , it's my pleasure to share
> > these
> > > information.
> > > We use StripeCompactionPolicy in almost all of our production clusters.
> > > And we let recently data in memstore flush to L0, limit stripe size to
> > > about 10G. Most of our regions are large than 50G, there even exist
> > regions
> > > larger than 2T... StripeCompactionPolicy has no major compactions, and
> it
> > > can limit compactions in only L0 and one stripe files, and it can
> perform
> > > cells deletion in one stripe just like the major-compaction. The
> pressure
> > > of compactions is broken down. Though the total files count in a region
> > > maybe a little larger than normal compactions, because the files are
> > > organized as in mini-regions, it works well for most read requests.
> > > And we also implemented a fast split and compact method based on
> > > StripeCompactionPolicy, practiced in all our production clusters,
> results
> > > show that split is very light weight and no need to perform read+write
> > > files compactions right after split. Details are in HBASE-25302
> > > , hope you have
> > > interest...
> > > Thanks.
> >
> >
> > The original link:
> > https://github.com/apache/hbase/pull/3152#issuecomment-824166990
> >
>

Re: [DISCUSS] Next 2.x releases

[Nick Dimiduk]

On Fri, Apr 30, 2021 at 5:40 PM 张铎(Duo Zhang)  wrote:

> Does anyone have interest in backporting HBASE-22120 to branch-2?
>

Yes, I think there would be interest in getting your tracing effort onto
branch-2 ; there's quite a few "watchers" on that Jira.

What about the backup work? Has it stabilized enough for backport?

Conversely, if we don't have a killer new feature for 2.5, does that mean
it's time for 3.0?

Andrew Purtell  于2021年5月1日周六 上午5:46写道：
>
> > Inline
> >
> > On Fri, Apr 30, 2021 at 2:11 PM Nick Dimiduk 
> wrote:
> >
> > > Heya,
> > >
> > > I'd like to have a planning discussion around our 2.x releases. 2.4
> seems
> > > to be humming along nicely, I think we won't have a need for 2.3 much
> > > longer. Likewise, it seems like it should be time to start planning
> 2.5,
> > > but when I look at the issues unique to that branch [0], I don't see
> the
> > > big new feature that justifies the new minor release. Rather, I see a
> > > number of items that should be backported to a 2.4.x. Do we have a big
> > new
> > > feature in the works? Should we consider backporting something from
> > master?
> > > Or maybe there's enough minor changes on 2.5 to justify the release...
> > but
> > > if so, which of them motivate users to upgrade?
> > >
> > > So, to summarize:
> > >
> > > - How much longer do we want to run 2.3?
> > > - What issues in the below query can be backported to 2.4? Any
> > volunteers?
> > >
> >
> > Thanks for starting this discussion, Nick.
> >
> > Looking at that report, issues like HBASE-24305, HBASE-25793, or
> > HBASE-25458 that clean up or reduce interfaces or refactor/move classes
> are
> > probably excluded from a patch release by our guidelines. Conversely,
> they
> > would not provide much value if backported.
> >
> > I also agree that the motivation for 2.5 here is thin as of now.
> Refactors,
> > interface improvements, or deprecation removals will be nice-to-haves
> when
> > there is a 2.5 someday.
> >
> > All the others in the report are either operational improvements that
> would
> > be nice to backport or bug fixes that should be backported.
> >
> > There might be case by case issues with compatibility during the pick
> > backs, but we can deal with them one at a time.
> >
> > If you are looking for a volunteer to do this, as 2.4 RM I am game.
> >
> >
> > - What's the big new feature that motivates 2.5?
> > >
> > > Thanks,
> > > Nick
> > >
> > > [0]:
> > >
> > >
> >
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20resolution%20%3D%20Fixed%20AND%20fixVersion%20%3D%202.5.0%20AND%20fixVersion%20not%20in%20(2.4.0%2C%202.4.1%2C%202.4.2%2C%202.4.3)%20ORDER%20BY%20priority%20DESC%2C%20updated%20DESC
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Words like orphans lost among the crosstalk, meaning torn from truth's
> > decrepit hands
> >- A23, Crosstalk
> >
>

[jira] [Resolved] (HBASE-25845) [branch-1] Precommit fails to build docker due to python-dateutil

[Reid Chan (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-25845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Reid Chan resolved HBASE-25845.
---
Fix Version/s: 1.7.0
   Resolution: Fixed

> [branch-1] Precommit fails to build docker due to python-dateutil
> -
>
> Key: HBASE-25845
> URL: https://issues.apache.org/jira/browse/HBASE-25845
> Project: HBase
>  Issue Type: Task
>Reporter: Reid Chan
>Assignee: Reid Chan
>Priority: Major
> Fix For: 1.7.0
>
>
> 02:34:22  Downloading/unpacking python-dateutil
> 02:34:22Cannot fetch index base URL https://pypi.python.org/simple/
> 02:34:22Could not find any downloads that satisfy the requirement 
> python-dateutil
> 02:34:22  Cleaning up...
> 02:34:22  No distributions at all found for python-dateutil
> 02:34:22  Storing debug log for failure in /root/.pip/pip.log
> 02:34:22  The command '/bin/sh -c pip install python-dateutil' returned a 
> non-zero code: 1
> 02:34:22  ERROR: Docker failed to build yetus/hbase:edccfe439a.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)