Re: Proposal to change meta operation timeout default - HBASE-28608

[Nick Dimiduk]

By compatibility, I just meant review this doc and think about how it
impacts your change.

https://hbase.apache.org/book.html#hbase.versioning

I do have an open TODO item for running API compat on PRs, but we're
not there yet.

On Thu, Nov 7, 2024 at 11:28 AM Daniel Roudnitsky (BLOOMBERG/ 919 3RD
A)  wrote:
>
> Thank you Duo and Nick for your replies, I will create a release note and a 
> backport PR, some quick process questions here:
> 1. For the branch-2 backport for targeting 2.7+, it is not a clean 
> backport/cherry-pick, should the backport PR be tied to a separate jira 
> issue? I have seen this being done in the project.
> 2. Is compatibility check just a matter of running checkcompatibility.py?
>
> I can also add the answers to these two questions to the developer guide in 
> the HBase reference if we feel that is appropriate.
>
> Thank you,
> Daniel
>
> From: dev@hbase.apache.org At: 11/07/24 01:18:05 UTC-5:00To:  
> dev@hbase.apache.org
> Subject: Re: Proposal to change meta operation timeout default - HBASE-28608
>
> I think this could also go into branch-2, but not branch-2.6 since it
> introduces some behavior changes.
>
> Please also fill the release note.
>
> I will merge this soon.
>
> Thanks for your patience Daniel.
>
> Nick Dimiduk  于2024年11月7日周四 02:42写道：
> >
> > Hi Daniel,
> >
> > I haven’t had much time for community work lately. You’re on my todo list.
> > I don’t remember the details of this PR, but you can stay ahead of the devs
> > by checking the API compat and determine if the change will backport to
> > branch-2. If so, make sure your work back-ports via simply cherry-pick, and
> > if not, post up a backport PR.
> >
> > I’ll try to get to your PR before the end of the week.
> >
> > Thanks,
> > Nick
> >
> > On Wed, 6 Nov 2024 at 12:51, Daniel Roudnitsky (BLOOMBERG/ 919 3RD A) <
> > droudnits...@bloomberg.net> wrote:
> >
> > > Hello Committers,
> > >
> > > The proposal received no concerns and has 3 (scattered) +1s:
> > > +1 in the thread below
> > > +1 in the PR
> > > https://github.com/apache/hbase/pull/6000#issuecomment-2183737439
> > > +1 in the jira issue
> > >
> https://issues.apache.org/jira/browse/HBASE-28608?focusedCommentId=17886921&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-
> 17886921
> > >
> > > Thank you to everyone for their feedback and review, given the support and
> > > lack of any concerns voiced, with the PR already approved, can this be
> > > committed, or is there anything else I can do to get this change 
> > > committed?
> > >
> > > Thank you,
> > > Daniel
> > >
> > > From: dev@hbase.apache.org At: 10/14/24 11:33:05 UTC-4:00To:
> > > dev@hbase.apache.org
> > > Subject: Re: Proposal to change meta operation timeout default -
> > > HBASE-28608
> > >
> > > Thank you Nick, Ray, Duo for feedback on the proposal, the current state
> > > is
> > > that it has three +1s and PR is open and approved -
> > > https://github.com/apache/hbase/pull/6000
> > >
> > > From: dev@hbase.apache.org At: 10/08/24 06:41:32 UTC-4:00To:
> > > dev@hbase.apache.org
> > > Subject: Re: Proposal to change meta operation timeout default -
> > > HBASE-28608
> > >
> > > Hi Daniel,
> > >
> > > I think that this change makes sense -- an unconfigured client meta
> > > request
> > > timeout should fall-back to the client request timeout -- however it is
> > > configured.
> > >
> > > Thanks,
> > > Nick
> > >
> > > On 2024/10/01 20:15:15 "Daniel Roudnitsky (BLOOMBERG/ 919 3RD A)" wrote:
> > > > Hello everyone,
> > > >
> > > > In HBASE-28608 I proposed a change to the default meta operation
> > > timeout, a
> > > behavioral change to the client. As its a behavioral change to the client
> > > I was
> > > suggested to start a dev list thread to get more opinions on the proposal
> > > (thank you Duo for suggestion and review). I would appreciate any reviews
> > > on
> > > this issue - https://issues.apache.org/jira/browse/HBASE-28608
> > > >
> > > > Thank you all,
> > > > Daniel Roudnitsky
> > >
> > >
> > >
>
>

Re: Proposal to change meta operation timeout default - HBASE-28608

[Nick Dimiduk]

Hi Daniel,

I haven’t had much time for community work lately. You’re on my todo list.
I don’t remember the details of this PR, but you can stay ahead of the devs
by checking the API compat and determine if the change will backport to
branch-2. If so, make sure your work back-ports via simply cherry-pick, and
if not, post up a backport PR.

I’ll try to get to your PR before the end of the week.

Thanks,
Nick

On Wed, 6 Nov 2024 at 12:51, Daniel Roudnitsky (BLOOMBERG/ 919 3RD A) <
droudnits...@bloomberg.net> wrote:

> Hello Committers,
>
> The proposal received no concerns and has 3 (scattered) +1s:
> +1 in the thread below
> +1 in the PR
> https://github.com/apache/hbase/pull/6000#issuecomment-2183737439
> +1 in the jira issue
> https://issues.apache.org/jira/browse/HBASE-28608?focusedCommentId=17886921&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17886921
>
> Thank you to everyone for their feedback and review, given the support and
> lack of any concerns voiced, with the PR already approved, can this be
> committed, or is there anything else I can do to get this change committed?
>
> Thank you,
> Daniel
>
> From: dev@hbase.apache.org At: 10/14/24 11:33:05 UTC-4:00To:
> dev@hbase.apache.org
> Subject: Re: Proposal to change meta operation timeout default -
> HBASE-28608
>
> Thank you Nick, Ray, Duo for feedback on the proposal, the current state
> is
> that it has three +1s and PR is open and approved -
> https://github.com/apache/hbase/pull/6000
>
> From: dev@hbase.apache.org At: 10/08/24 06:41:32 UTC-4:00To:
> dev@hbase.apache.org
> Subject: Re: Proposal to change meta operation timeout default -
> HBASE-28608
>
> Hi Daniel,
>
> I think that this change makes sense -- an unconfigured client meta
> request
> timeout should fall-back to the client request timeout -- however it is
> configured.
>
> Thanks,
> Nick
>
> On 2024/10/01 20:15:15 "Daniel Roudnitsky (BLOOMBERG/ 919 3RD A)" wrote:
> > Hello everyone,
> >
> > In HBASE-28608 I proposed a change to the default meta operation
> timeout, a
> behavioral change to the client. As its a behavioral change to the client
> I was
> suggested to start a dev list thread to get more opinions on the proposal
> (thank you Duo for suggestion and review). I would appreciate any reviews
> on
> this issue - https://issues.apache.org/jira/browse/HBASE-28608
> >
> > Thank you all,
> > Daniel Roudnitsky
>
>
>

[DISCUSS] Upper bound on RPC memory usage

[Nick Dimiduk]

Heya team,

We hit some production troubles pertaining to clients sending very
large multi-gets. Even with otherwise reasonable cell- and row-size
limits, even with maximum multi-action sizes in place, even with QoS
and our fancy IO-based Quotas, the pressure was enough to push over a
region server or three. It got me thinking that we need some kind of
pressure gauge in the RPC layer that can protect the RS. This wouldn't
be a QoS or Quota kind of feature, it's not about fairness between
tenants, rather it's a safety mechanism, a kind of pressure valve. I
wonder if something like this already exists or maybe you know of a
ticket already filed with some existing discussion.

My napkin-sketch is something like a metric that tracks the amount of
heap size consumed by active request and response objects. When the
metrics hits a limit, we start to reject new requests with a retryable
exception. I don't know if we want the overhead of tracking this value
exactly, so maybe the value is populated only by new requests and then
we have some crude mechanism of decay. Does Netty already have
something like this? I'd say this is in lieu of an actual streaming
RPC harness, but I think even a streaming system would benefit from
such a backpressure strategy.

It occurs to me that I don't know the current state of active memory
tracking in the region server. I recall there was some work to make
memstore and blockcache resize dynamically. Maybe this new system adds
a 3rd component to the computation.

Thoughts? Ideas?

Thanks,
Nick

Re: [DISCUSS] Separating Web UI code into separate module(s) and Jar(s)

[Nick Dimiduk]

I am in favor of splitting our UI from our http APIs. Our inheritance
from Hadoop has served to expedite, but also complicated this aspect.
Having the web dependencies in mapreduce has also caused classpath
conflicts over the years. It also might be interesting to have
separate thread pools for UI handlers vs. API handlers.

A while back I introduced a new pattern of using a (relatively) modern
jersey for webservice endpoints. I don't know if adopting that further
would help in this effort. As I recall, it achieved some of the
modularity that you describe, though it all resides its own package in
the same hbase-server jar.

On Fri, Oct 25, 2024 at 8:38 AM Istvan Toth  wrote:
>
> Currently the Master and RegionServer modules include both the RPC server
> functionality and the WebUI (ant JMX and metrics) HTTP server code.
>
> My problem with this is that we do include these in hbase-server.jar, and
> as such they are also inherited by the hbase mapreduce classpath, which is
> bloat and has possible CVE implications.
> (Or at the very least we include code that we don't have half the
> libraries for on the classpath)
>
> What is your opinion on splitting out the HTTP functionality into separate
> module(s), and separate JAR(s) from hbase-server ?
>
> It's a bit tricky, since we start the RPC and HTTP servers together, so
> we'd either have to move the main and startup classes to the HTTP module,
> or separate the startup code.
>
> Another way to look at this is to separate the code needed by integrations
> and the code needed for actually running network services, in which case
> even the actual RPC server initialization could be split out.
>
> Is this even possible, or do I miss dependencies ?
> If possible, is this something you see value in ?
> I don't have the bandwidth to work on this short term but Nihal's recent
> work has got me thinking about this topic.
> (I'm thinking strictly about hbase-server, not the REST of Thrift servers,
> where http is integral)
>
> Istvan

Re: [DISCUSS] Using Hadoop 3.3.6 and 3.4.0 as default versions

[Nick Dimiduk]

On Mon, Oct 28, 2024 at 11:00 AM Istvan Toth  wrote:
>
> I have looked at branch-2.5, but the nightly looks off there, as it runs
> the packaging tests with Hadoop 3.1.1, which it  doesn't even officially
> support anymore.
>
> What should we do with branch-2.5 ?
>
> I think that it would not be a lot of extra  work to backport everything,
> both the backwards compatibility tests and defaulting Hadoop to 3.4.1.
> We just have to update the version in the pom, and add 3.2.4 to the list of
> versions to test for backwards compatibility and integration (and remove
> 3.1.1).
>
> I would prefer to have uniform tests and default to Hadoop 3.4.1 on all
> active branches.
> Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x support may
> be useful for users for migration and CVE mitigation purposes.
>
> WDYT ?

branch-2.5's default hadoop3 version is 3.2.4. That's a big dependency
to change for a patch release. I don't think that we can get away with
that change and maintain our compatibility obligations. I'm not up to
speed on the current state of CVEs for this older (EOL?) version, so
we have that dimension to consider. If the newer version is "drop-in"
compatible (and only if), then I have no issue with moving that
release line forward. Ultimately it's the release manager for 2.5 to
make a determination, so I defer to Andrew's assessment.

I am in favor of backing-porting the improved testing coverage you've
added to branch-2.5. It would be great to understand if branch-2.5 (1)
compiled against 3.2.4 will run on Hadoop 3.4.1 and (2) builds and
tests out on 3.4.1. That will give the more security-minded users
additional confidence in bumping their hadoop dependency on their own.

Thanks,
Nick

> On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth  wrote:
>
> > We could also move the default to 3.4.1 directly.
> > We already test for 3.4.0 in the nightly job.
> >
> > On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) 
> > wrote:
> >
> >> And seems hadoop 3.4.1 is out. we could see whether to bump to this
> >> version later?
> >>
> >> Istvan Toth  于2024年10月21日周一 20:56写道：
> >> >
> >> > I have merged the new tests to the nightly Jenkins runs on master.
> >> >
> >> > They have identified another 3.4.0 incompatibility:
> >> > HBASE-28929 
> >> >
> >> > I will hold off backporting the test changes until HBASE-28929 is
> >> resolved.
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: st...@cloudera.com
> > cloudera.com 
> > [image: Cloudera] 
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> > Cloudera on LinkedIn] 
> > --
> > --
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: st...@cloudera.com
> cloudera.com 
> [image: Cloudera] 
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
> --

Re: [DISCUSS] Using Hadoop 3.3.6 and 3.4.0 as default versions

[Nick Dimiduk]

Hi Istvan,

Thank you so much for continuing to dig in here.

We do save off test run logs, they are too big for jenkins so we ship
them off to nightlies.a.o. For example, you can find logs from the
last nightly run of the master branch on
https://nightlies.apache.org/hbase/HBase%20Nightly/master/1194/. I
haven't cracked open one of these archives in a while, so I don't know
if the logs you need will be present, but maybe it's worth taking a
look.

Thanks,
Nick

On Tue, Oct 29, 2024 at 8:35 AM Istvan Toth  wrote:
>
> Looking at the nightly tests, the test stability has decreased.
> AFAICT there is not much of a pattern to the test failures, and most of the
> failures seem to be
> minicluster startup failures.
> Some of the failures are for Hadoop2, and Hadoop 3.3.5 and 3.3.6 seems as
> prone to the failures as
>  3.4.0 and 3.4.1, so I don't think these are directly caused by the upgrade.
> Some of this may be explained by us running much more tests now, and thus
> having a higher chance of triggering
> random/transient errors in them.
> Unfortunately, debugging the minicluster failures would require the full
> debug log from the failing tests, which none of the jobs save.
>
>
>
> On Mon, Oct 28, 2024 at 10:58 AM Istvan Toth  wrote:
>
> > I have backported the compatibility checks into the nightlies down to
> > branch-2.6.
> > As soon as the tests are stable, I will also backport the change for
> > bumping the  default version to 3.4.1 in branch-2.6 and later.
> >
> > I have looked at branch-2.5, but the nightly looks off there, as it runs
> > the packaging tests with Hadoop 3.1.1, which it  doesn't even officially
> > support anymore.
> >
> > What should we do with branch-2.5 ?
> >
> > I think that it would not be a lot of extra  work to backport everything,
> > both the backwards compatibility tests and defaulting Hadoop to 3.4.1.
> > We just have to update the version in the pom, and add 3.2.4 to the list
> > of versions to test for backwards compatibility and integration (and remove
> > 3.1.1).
> >
> > I would prefer to have uniform tests and default to Hadoop 3.4.1 on all
> > active branches.
> > Having a (few) final 2.5.x release(s) with tested Hadoop 3.4.x support may
> > be useful for users for migration and CVE mitigation purposes.
> >
> > WDYT ?
> >
> > On Mon, Oct 21, 2024 at 6:54 PM Istvan Toth  wrote:
> >
> >> We could also move the default to 3.4.1 directly.
> >> We already test for 3.4.0 in the nightly job.
> >>
> >> On Mon, Oct 21, 2024 at 3:49 PM 张铎(Duo Zhang) 
> >> wrote:
> >>
> >>> And seems hadoop 3.4.1 is out. we could see whether to bump to this
> >>> version later?
> >>>
> >>> Istvan Toth  于2024年10月21日周一 20:56写道：
> >>> >
> >>> > I have merged the new tests to the nightly Jenkins runs on master.
> >>> >
> >>> > They have identified another 3.4.0 incompatibility:
> >>> > HBASE-28929 
> >>> >
> >>> > I will hold off backporting the test changes until HBASE-28929 is
> >>> resolved.
> >>>
> >>
> >>
> >> --
> >> *István Tóth* | Sr. Staff Software Engineer
> >> *Email*: st...@cloudera.com
> >> cloudera.com 
> >> [image: Cloudera] 
> >> [image: Cloudera on Twitter]  [image:
> >> Cloudera on Facebook]  [image:
> >> Cloudera on LinkedIn] 
> >> --
> >> --
> >>
> >
> >
> > --
> > *István Tóth* | Sr. Staff Software Engineer
> > *Email*: st...@cloudera.com
> > cloudera.com 
> > [image: Cloudera] 
> > [image: Cloudera on Twitter]  [image:
> > Cloudera on Facebook]  [image:
> > Cloudera on LinkedIn] 
> > --
> > --
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: st...@cloudera.com
> cloudera.com 
> [image: Cloudera] 
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
> --

[jira] [Resolved] (HBASE-28847) Release 2.6.1

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28847?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28847.
--
Resolution: Done

> Release 2.6.1
> -
>
> Key: HBASE-28847
> URL: https://issues.apache.org/jira/browse/HBASE-28847
> Project: HBase
>  Issue Type: Task
>  Components: community
>    Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> We have 130+ commits on branch-2.6 ; we're over-due for 2.6.1. That's a lot 
> of changes, so lets track this a little more carefully than we might a 
> standard patch release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28925) Add announcement as a release in GitHub

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28925?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28925.
--
Resolution: Done

> Add announcement as a release in GitHub
> ---
>
> Key: HBASE-28925
> URL: https://issues.apache.org/jira/browse/HBASE-28925
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28857) Send announce email

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28857?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28857.
--
Resolution: Done

> Send announce email
> ---
>
> Key: HBASE-28857
> URL: https://issues.apache.org/jira/browse/HBASE-28857
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[ANNOUNCE] Apache HBase 2.6.1 is now available for download

[Nick Dimiduk]

The HBase team is happy to announce the immediate availability of HBase
2.6.1.

Apache HBase™ is an open-source, distributed, versioned, non-relational
database.
Apache HBase gives you low latency random access to billions of rows with
millions of columns atop non-specialized hardware. To learn more about
HBase,
see https://hbase.apache.org/.

HBase 2.6.1 is the first minor release in the HBase 2.6.x line, which aims
to
improve the stability and reliability of HBase. This release includes
roughly
130 resolved issues not covered by previous 2.x releases. This means that
it's
particularly large as HBase patch releases go.

Notable improvements include:
- A new Dual File Compaction algorithm in HBASE-25972
- Reliability improvements related to Master procedure execution
- Bug fixes and enhancements to the new Backup & Restore feature
- Bug fixes and improvements throughout the system

The full list of issues can be found in the included CHANGES.md and
RELEASENOTES.md,
or via our issue tracker:

https://s.apache.org/hbase-2.6.1-jira

To download please follow the links and instructions on our website:

https://hbase.apache.org/downloads.html


Questions, comments, and problems are always welcome at:
dev@hbase.apache.org.

Thanks to all who contributed and made this release possible.

Cheers,
The HBase Dev Team

[jira] [Resolved] (HBASE-28853) Publish staged repository in Nexus

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28853.
--
Resolution: Done

> Publish staged repository in Nexus
> --
>
> Key: HBASE-28853
> URL: https://issues.apache.org/jira/browse/HBASE-28853
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28926) Publish release artifacts in SVN

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28926.
--
Resolution: Done

> Publish release artifacts in SVN
> 
>
> Key: HBASE-28926
> URL: https://issues.apache.org/jira/browse/HBASE-28926
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>
> Don't forget to also drop the old release candidate artifacts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28926) Publish release artifacts in SVN

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28926:


 Summary: Publish release artifacts in SVN
 Key: HBASE-28926
 URL: https://issues.apache.org/jira/browse/HBASE-28926
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk


Don't forget to also drop the old release candidate artifacts.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28925) Add announcement as a release in GitHub

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28925:


 Summary: Add announcement as a release in GitHub
 Key: HBASE-28925
 URL: https://issues.apache.org/jira/browse/HBASE-28925
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28856) Update reporter tool with new release

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28856.
--
Resolution: Done

https://reporter.apache.org/addrelease.html?hbase

> Update reporter tool with new release
> -
>
> Key: HBASE-28856
> URL: https://issues.apache.org/jira/browse/HBASE-28856
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28855) Push signed release tag to git

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28855.
--
Resolution: Done

> Push signed release tag to git
> --
>
> Key: HBASE-28855
> URL: https://issues.apache.org/jira/browse/HBASE-28855
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28854) Release version 2.6.1 in Jira

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28854?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28854.
--
Resolution: Done

> Release version 2.6.1 in Jira
> -
>
> Key: HBASE-28854
> URL: https://issues.apache.org/jira/browse/HBASE-28854
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28855) Push signed release tag to git

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28855:
--

> Push signed release tag to git
> --
>
> Key: HBASE-28855
> URL: https://issues.apache.org/jira/browse/HBASE-28855
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28855) Push signed release tag to git

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28855?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28855.
--
Resolution: Fixed

git tag -s rel/2.6.1 2.6.1RC1^{} -u 0x1156CF920BB24B8A -m "Apache HBase release 
2.6.1"

> Push signed release tag to git
> --
>
> Key: HBASE-28855
> URL: https://issues.apache.org/jira/browse/HBASE-28855
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28849) Review open issues and mark likely inclusion candidates with fixVersion=2.6.1

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28849.
--
Resolution: Done

> Review open issues and mark likely inclusion candidates with fixVersion=2.6.1
> -
>
> Key: HBASE-28849
> URL: https://issues.apache.org/jira/browse/HBASE-28849
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28848) Audit Jira vs. git commit history

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28848?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28848.
--
Resolution: Done

Thanks [~zhangduo]!

> Audit Jira vs. git commit history
> -
>
> Key: HBASE-28848
> URL: https://issues.apache.org/jira/browse/HBASE-28848
> Project: HBase
>  Issue Type: Sub-task
>  Components: community
>    Reporter: Nick Dimiduk
>Assignee: Duo Zhang
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28852) Propose Release Candidate(s)

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28852?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28852.
--
  Assignee: Nick Dimiduk
Resolution: Done

> Propose Release Candidate(s)
> 
>
> Key: HBASE-28852
> URL: https://issues.apache.org/jira/browse/HBASE-28852
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28851) Run a correctness test with ITBLL

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28851?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28851.
--
Resolution: Done

> Run a correctness test with ITBLL
> -
>
> Key: HBASE-28851
> URL: https://issues.apache.org/jira/browse/HBASE-28851
> Project: HBase
>  Issue Type: Sub-task
>        Reporter: Nick Dimiduk
>    Assignee: Nick Dimiduk
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

With 3 binding +1 votes and no other ballots cast, this measure passes. Let
me proceed with finalizing the release.

Thanks to all who contributed and to all who voted.

On Wed, 16 Oct 2024 at 06:52, Pankaj Kumar  wrote:

>  +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (17.0.11): ok
>  - mvn clean apache-rat:check -D hadoop.profile=3.0
> * Built from source (17.0.11): ok
>  - mvn clean install -D hadoop.profile=3.0 -DskipTests
> * Unit tests pass (17.0.11): ok
>  - mvn clean package -P runSmallTests -D hadoop.profile=3.0
> -Dsurefire.rerunFailingTestsCount=3
>
> - Run basic hbase shell commands, works fine
> - Loaded 20M records using YCSB, nothing unusual
> - HBase web UI looks good.
> - Checked API Compatibility report, 100% compatible
>
> Regards,
> Pankaj
>
> On Tue, Oct 15, 2024 at 9:31 PM Nick Dimiduk  wrote:
>
> > And my +1
> >
> > * Signature: ok
> > * Checksum : ok
> > * Rat check (17.0.11): ok
> >  - mvn clean apache-rat:check -D hadoop.profile=3.0 -D
> > skipTests=true
> > * Built from source (1.8.0_422): ok
> >  - mvn clean install -DskipTests
> > * Built from source (11.0.24): ok
> >  - mvn clean install -D hadoop.profile=3.0 -DskipTests
> > * Built from source (17.0.11): ok
> >  - mvn clean install -D skipTests=true -DskipTests
> > * Built from source (21.0.4): ok
> >  - mvn clean install -D hadoop.profile=3.0 -DskipTests
> > * Previously mentioned ITBLL job: verified 250m links on a small
> > cluster with chaos. It was running JDK21 + Hadoop-3.3.6.
> > * Checked the API Compatibilty report vs. Public and
> LimitedPrivate
> > * Nightly Jenkins builds have been no worse than has become our
> > usual.
> >
> > On Wed, Oct 9, 2024 at 2:22 PM Nick Dimiduk  wrote:
> >
> > > Please vote on this Apache hbase release candidate,
> > > hbase-2.6.1RC1
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.6.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.6.1RC1:
> > >
> > >   https://github.com/apache/hbase/tree/2.6.1RC1
> > >
> > > This tag currently points to git reference
> > >
> > >   7ed50b4dd742269a78875fb32112215f831284ff
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > >   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1560/
> > >
> > > Maven artifacts for hadoop3 are available in a staging repository at:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1561/
> > >
> > > Artifacts were signed with the 0xEF4EBF27 key which can be found in:
> > >
> > >   https://downloads.apache.org/hbase/KEYS
> > >
> > > To learn more about Apache hbase, please see
> > >
> > >   http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
>

[jira] [Reopened] (HBASE-28897) Force CF compatibility during incremental backup

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28897:
--

Re-opening due to api incompatibility issue reported on the master PR.

>  Force CF compatibility during incremental backup
> -
>
> Key: HBASE-28897
> URL: https://issues.apache.org/jira/browse/HBASE-28897
> Project: HBase
>  Issue Type: Bug
>  Components: backup&restore
>Affects Versions: 3.0.0-beta-1, 4.0.0-alpha-1, 2.7.0, 2.6.2
>Reporter: Hernan Gelaf-Romer
>Assignee: Hernan Gelaf-Romer
>Priority: Major
>  Labels: pull-request-available
>
> Incremental backups can be taken even if the table descriptor of the current 
> table does not match the column families of the full backup for that same 
> table. When restoring the table, we choose to use the families of the full 
> backup. This can cause the restore process to fail if we add a column family 
> in the incremental backup that doesn't exist in the full backup. The bulkload 
> process will fail because it is trying to write column families that don't 
> exist in the restore table. 
>  
> I think the correct solution here is to prevent incremental backups from 
> being taken if the families of the current table don't match those of the 
> full backup. This will force users to instead take a full backup.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

And my +1

* Signature: ok
* Checksum : ok
* Rat check (17.0.11): ok
 - mvn clean apache-rat:check -D hadoop.profile=3.0 -D
skipTests=true
* Built from source (1.8.0_422): ok
 - mvn clean install -DskipTests
* Built from source (11.0.24): ok
 - mvn clean install -D hadoop.profile=3.0 -DskipTests
* Built from source (17.0.11): ok
 - mvn clean install -D skipTests=true -DskipTests
* Built from source (21.0.4): ok
 - mvn clean install -D hadoop.profile=3.0 -DskipTests
* Previously mentioned ITBLL job: verified 250m links on a small
cluster with chaos. It was running JDK21 + Hadoop-3.3.6.
* Checked the API Compatibilty report vs. Public and LimitedPrivate
* Nightly Jenkins builds have been no worse than has become our
usual.

On Wed, Oct 9, 2024 at 2:22 PM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC1:
>
>   https://github.com/apache/hbase/tree/2.6.1RC1
>
> This tag currently points to git reference
>
>   7ed50b4dd742269a78875fb32112215f831284ff
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1560/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1561/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I ran a small ITBLL job against rc1 on a cluster of 9 region servers and 3
masters, generating 250m nodes, using slowDeterministic monkey. The job
took 9 hours and verify succeeded.

On Wed, Oct 9, 2024 at 2:22 PM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC1:
>
>   https://github.com/apache/hbase/tree/2.6.1RC1
>
> This tag currently points to git reference
>
>   7ed50b4dd742269a78875fb32112215f831284ff
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1560/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1561/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

Thanks for looking at this.

I checked the CI history for the branch before the release and it looked
okay. Looking again today, it seems a bit worse, but I don't see any
hbase-shell failures when I look through the recent failed test runs.

Running locally, all the tests in hbase-shell pass for me. TestTableShell
is among the tests run.

$ JAVA_HOME=$JAVA17_HOME mvn clean install -Dhadoop.profile=3.0 -T1.0C -am
-pl hbase-shell -Dmaven.repo.local=$HOME/tmp/maven-repo-local
-DskipTests=true && JAVA_HOME=$JAVA17_HOME mvn test -Dhadoop.profile=3.0
-pl hbase-shell -Dmaven.repo.local=$HOME/tmp/maven-repo-local
...
[INFO] Results:
[INFO]
[INFO] Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO]

[INFO] BUILD SUCCESS
[INFO]


On Mon, Oct 14, 2024 at 4:40 AM 张铎(Duo Zhang)  wrote:

> I'm running UTs for this RC.
>
> The failures in hbase-server seemed OK, some flakies ones but they
> passed when rerun.
> TestAcidGuaranteesWithEagerPolicy failed but it is known to be heavy and
> flaky.
> TestMutualTlsClientSideNonLocalhost seemed to be my environment problem.
>
> But there are bunch of failures in hbase-shell module, and I manually
> tried running TestTableShell
>
> org/jruby/RubyKernel.java:1233:in `catch'
> org/jruby/RubyKernel.java:1238:in `catch'
> org/jruby/RubyKernel.java:1233:in `catch'
>   256:   # Put two columns with colons in their qualifiers
>   257:   @test_table.put(rowkey, col1,
> org.apache.hadoop.hbase.util.Bytes.toBytes(1))
>   258:   @test_table.put(rowkey, col2,
> org.apache.hadoop.hbase.util.Bytes.toBytes(2))
>   ^[[48;5;16;38;5;226;1m  => 259:   assert_equal(2,
> @test_table._get_internal(rowkey).length)^[[0m
>   260:
>   261:   # Increment the second column by 10 => 2 + 10 => 12
>   262:   @test_table.incr(rowkey, col2, 10)
> src/test/ruby/hbase/table_test.rb:259:in `block in
> test_should_work_with_qualifiers_with_colons'
> Error:
> ^[[48;5;16;38;5;226;1mtest_should_work_with_qualifiers_with_colons(Hbase::TableSimpleMethodsTest)^[[0m:
> NoMethodError: undefined method `length' for nil:NilClass
>
> Seemed to be real problem or at least test issue?
>
> Nick Dimiduk  于2024年10月9日周三 20:41写道：
> >
> > I have generated an API Compatibility report for this release
> > candidate that includes the LimitedPrivate annotated classes. It is
> > available for your consideration at
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/api_compare_2.6.0_to_2.6.1RC1_Public_LimitedPrivate.html
> >
> > Thanks,
> > Nick
> >
> > On Wed, Oct 9, 2024 at 2:22 PM Nick Dimiduk  wrote:
> >
> > > Please vote on this Apache hbase release candidate,
> > > hbase-2.6.1RC1
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.6.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.6.1RC1:
> > >
> > >   https://github.com/apache/hbase/tree/2.6.1RC1
> > >
> > > This tag currently points to git reference
> > >
> > >   7ed50b4dd742269a78875fb32112215f831284ff
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > >   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1560/
> > >
> > > Maven artifacts for hadoop3 are available in a staging repository at:
> > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1561/
> > >
> > > Artifacts were signed with the 0xEF4EBF27 key which can be found in:
> > >
> > >   https://downloads.apache.org/hbase/KEYS
> > >
> > > To learn more about Apache hbase, please see
> > >
> > >   http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
>

Re: [VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I have generated an API Compatibility report for this release
candidate that includes the LimitedPrivate annotated classes. It is
available for your consideration at
https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/api_compare_2.6.0_to_2.6.1RC1_Public_LimitedPrivate.html

Thanks,
Nick

On Wed, Oct 9, 2024 at 2:22 PM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC1:
>
>   https://github.com/apache/hbase/tree/2.6.1RC1
>
> This tag currently points to git reference
>
>   7ed50b4dd742269a78875fb32112215f831284ff
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1560/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1561/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

This VOTE is canceled. Please see the second release candidate for your
consideration.

Thanks,
Nick

On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC0:
>
>   https://github.com/apache/hbase/tree/2.6.1RC0
>
> This tag currently points to git reference
>
>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[VOTE] The second release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

Please vote on this Apache hbase release candidate,
hbase-2.6.1RC1

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 2.6.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.6.1RC1:

  https://github.com/apache/hbase/tree/2.6.1RC1

This tag currently points to git reference

  7ed50b4dd742269a78875fb32112215f831284ff

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC1/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1560/

Maven artifacts for hadoop3 are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1561/

Artifacts were signed with the 0xEF4EBF27 key which can be found in:

  https://downloads.apache.org/hbase/KEYS

To learn more about Apache hbase, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI

[Nick Dimiduk]

Ah, thanks Duo, now I understand you.

I see your point about how Github Issues could help to enhance the
community. I didn't realize that email has become so old-fashioned, but
then one of my colleagues just created an internal channel that imports
mail to this dev list...

I'm personally fine with adopting more features of Github, despite very
real concerns from Apache about self-hosting. Should the HBase developer
community be beholden to the whims of any company? However, short of ASF
infra beefing up its capabilities, or we the HBase developer community
taking infrastructure into our own hands (I think we can get VMs
provisioned...), adopting more capabilities out of Github does seem like a
practical path forward. What I don't want is multiple places of authority
for reporting bugs, tracking changes, managing releases.

I think this is a fruitful discussion but has wondered a bit off topic.
Might I suggest that we split this off to a separate discussion? I'd like
to see how folks are thinking specifically around CI.

Thanks,
Nick

On Wed, Oct 9, 2024 at 5:04 AM 张铎(Duo Zhang)  wrote:

> About github issues, I'm only talking about how to enlarge the
> community, not about other things. For now using github issues and
> github actions are independent, unless we plan to move off from jira
> too in the future.
>
> To be honest, most developers in China today do not use email any
> more, enabling github issues will give them a good place to ask
> questions and discuss things. We do not need to change our current
> workflow, we still use jira for issue management.
> There are Apache projects that use github issues for issue management
> so I think there are ways to not break the ASF policy when using
> github issues, like sending all things to issues at a.o?
>
> Thanks.
>
> Istvan Toth  于2024年10月8日周二 18:08写道：
> >
> > I'm not sure about enabling github issues.
> >
> > We already have the mailing lists, JIRA, and the pull requests to keep
> > track of, I'm afraid that adding another forum would overcomplicate
> things.
> >
> > IMO migrating to GH actions and using GH issues are independent from each
> > other.
> >
> > The current JIRA signup process is definitely bad, we are often supposed
> to
> > be making decisions on cutesy usernames without
> > any real context, and we cannot even ask for more information.
> >
> > Maybe we could add some kind of form to the docs where we list some
> > questions from ppl trying to sign up that would be too much work
> > for a spammer ?
> >
> > Istvan
> >
> >
> >
> >
> > On Tue, Oct 8, 2024 at 9:30 AM 张铎(Duo Zhang) 
> wrote:
> >
> > > Oh, typo, PMCs -> PMC members
> > >
> > > 张铎(Duo Zhang)  于2024年10月8日周二 11:46写道：
> > > >
> > > > For me I think we could first enable the github issues, for users to
> > > > ask questions and discuss things. And if there are actual bugs or
> > > > something which require code changes, we could file an jira, and also
> > > > let the contributor to register a jira account.
> > > > I think this is also easier for our PMCs to decide whether a jira
> > > > account is necessary for a given user comparing to the current
> > > > workflow. The private mailing list is full of jira registrations
> > > > notifications and hard to find other useful information...
> > > >
> > > > And on moving to github actions, in general I'm +1 on this. We should
> > > > try to follow modern ways.
> > > >
> > > > And on the funding side, we still have 10 machines, we could contact
> > > > INFRA to see how to make use of these machines if we switch to github
> > > > actions.
> > > >
> > > > Thanks.
> > > >
> > > > Istvan Toth  于2024年10月2日周三 17:31写道：
> > > > >
> > > > > I've been working on modifying the existing Jenkinsfile, and it has
> > > been a
> > > > > horrible experience, especially as I'm trying to mix declarative
> and
> > > > > scripted syntax.
> > > > > I think from a usability standpoint GH actions would be a win.
> > > > >
> > > > > On the other hand, our Jenkinsfiles don't do that much, as most of
> the
> > > > > actual CI process is performed via Yetus, so migration shouldn't
> be a
> > > huge
> > > > > amount of work.
> > > > >
> > > > > I seem to recall seeing similar discussions on ASF mailing lists,
> but I
> > > > 

[jira] [Resolved] (HBASE-28770) Support partial results in AggregateImplementation and AsyncAggregationClient

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28770.
--
Resolution: Fixed

> Support partial results in AggregateImplementation and AsyncAggregationClient
> -
>
> Key: HBASE-28770
> URL: https://issues.apache.org/jira/browse/HBASE-28770
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, Coprocessors, Quotas
>Affects Versions: 2.6.0
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Currently there is a gap in the coverage of HBase's quota-based workload 
> throttling. Requests sent by {{[Async]AggregationClient}} reach 
> {{AggregateImplementation}}. This then executes Scans in a way that bypasses 
> the quota system. We see issues with this at Hubspot where clusters suffer 
> under this load and we don't have a good way to protect them.
> In this ticket I'm teaching {{AggregateImplementation}} to optionally stop 
> scanning when a throttle is violated, and send back just the results it has 
> accumulated so far. In addition, it will send back a row key to 
> {{AsyncAggregationClient}}. When the client gets a response with a row key, 
> it will sleep in order to satisfy the throttle, and then send a new request 
> with a scan starting at that row key. This will have the effect of continuing 
> the work where the last request stopped.
> This feature will be unconditionally enabled by {{AsyncAggregationClient}} 
> once this ticket is finished. {{AggregateImplementation}} will not assume 
> that clients support partial results, however, so it can keep supporting 
> older clients. For clients that do not support partial results, throttles 
> will not be respecting, and results will always be complete.
> This feature was [first proposed on the mailing 
> list|https://lists.apache.org/thread/1vqnxb71z7swq2cogz4qg3cn6b10xp4v]. 
> Builds on work in HBASE-28346.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28842) TestRequestAttributes should fail when expected

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28842.
--
Resolution: Fixed

No adjustment needed. Resolving. 

> TestRequestAttributes should fail when expected
> ---
>
> Key: HBASE-28842
> URL: https://issues.apache.org/jira/browse/HBASE-28842
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Evelyn Boland
>Assignee: Evelyn Boland
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Problem:
> The tests in the TestRequestAttributes class pass even when they should fail. 
> I've included an example of a test that should fail but does not below.
> Fix:
> Throw an IOException in the AttributesCoprocessor when the map of expected 
> request attributes does not match the map of given request attributes.
>  
> Test: 
> We set 2+ request attributes on the Get request but always return 0 request 
> attributes from AttributesCoprocessor::getRequestAttributesForRowKey method. 
> Yet the test passes even though the map of expected request attributes never 
> matches the map of given request attributes.
> {code:java}
> @Category({ ClientTests.class, MediumTests.class })
> public class TestRequestAttributes {
>   @ClassRule
>   public static final HBaseClassTestRule CLASS_RULE =
> HBaseClassTestRule.forClass(TestRequestAttributes.class);
>   private static final byte[] ROW_KEY1 = Bytes.toBytes("1");
>   private static final Map> 
> ROW_KEY_TO_REQUEST_ATTRIBUTES =
> new HashMap<>();
>   static {
> CONNECTION_ATTRIBUTES.put("clientId", Bytes.toBytes("foo"));
> ROW_KEY_TO_REQUEST_ATTRIBUTES.put(ROW_KEY1, addRandomRequestAttributes());
>   }
>   private static final ExecutorService EXECUTOR_SERVICE = 
> Executors.newFixedThreadPool(100);
>   private static final byte[] FAMILY = Bytes.toBytes("0");
>   private static final TableName TABLE_NAME = 
> TableName.valueOf("testRequestAttributes");
>   private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
>   private static SingleProcessHBaseCluster cluster;
>   @BeforeClass
>   public static void setUp() throws Exception {
> cluster = TEST_UTIL.startMiniCluster(1);
> Table table = TEST_UTIL.createTable(TABLE_NAME, new byte[][] { FAMILY }, 
> 1,
>   HConstants.DEFAULT_BLOCKSIZE, AttributesCoprocessor.class.getName());
> table.close();
>   }
>   @AfterClass
>   public static void afterClass() throws Exception {
> cluster.close();
> TEST_UTIL.shutdownMiniCluster();
>   }
> @Test
> public void testRequestAttributesGet() throws IOException {
>   Configuration conf = TEST_UTIL.getConfiguration();
>   try (
> Connection conn = ConnectionFactory.createConnection(conf, null, 
> AuthUtil.loginClient(conf),
>   CONNECTION_ATTRIBUTES);
> Table table = configureRequestAttributes(conn.getTableBuilder(TABLE_NAME, 
> EXECUTOR_SERVICE),
>   ROW_KEY_TO_REQUEST_ATTRIBUTES.get(ROW_KEY1)).build()) {
> table.get(new Get(ROW_KEY1));
>   }
> }
> private static Map addRandomRequestAttributes() {
>   Map requestAttributes = new HashMap<>();
>   int j = Math.max(2, (int) (10 * Math.random()));
>   for (int i = 0; i < j; i++) {
> requestAttributes.put(String.valueOf(i), 
> Bytes.toBytes(UUID.randomUUID().toString()));
>   }
>   return requestAttributes;
> }
> public static class AttributesCoprocessor implements RegionObserver, 
> RegionCoprocessor {
> @Override
> public Optional getRegionObserver() {
>   return Optional.of(this);
> }
> @Override
> public void preGetOp(ObserverContext c, Get 
> get,
>   List result) throws IOException {
>
> validateRequestAttributes(getRequestAttributesForRowKey(get.getRow(;   
> }
> private Map getRequestAttributesForRowKey(byte[] rowKey) {
>   return Collections.emptyMap(); // This line helps demonstrate the bug
> }
> private boolean validateRequestAttributes(Map 
> requestAttributes) {
>   RpcCall rpcCall = RpcServer.getCurrentCall().get();
>   Map attrs = rpcCall.getRequestAttributes();
>   if (attrs.size() != requestAttributes.size()) {
> return;
>   }
>   for (Map.Entry attr : attrs.entrySet()) {
> if (!requestAttributes.containsKey(attr.getKey())) {
>   return;
> }
> if (!Arrays.equals(requestAttributes.get(attr.getKey()), 
> attr.getValue())) {
>   return;
> }
>   }
>   return;
> }
>   }
> } {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28396) Quota throttling can cause a leak of scanners

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28396.
--
Resolution: Fixed

Resolving. No adjustments needed.

> Quota throttling can cause a leak of scanners
> -
>
> Key: HBASE-28396
> URL: https://issues.apache.org/jira/browse/HBASE-28396
> Project: HBase
>  Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Hernan Gelaf-Romer
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> In RSRpcServices.scan, we check the quota after having created a new 
> RegionScannerHolder. If the quota is exceeded, an exception will be thrown. 
> In this case, we can't send the scannerName back to the client because it's 
> just an exception. So the client will be forced to retry the openScanner 
> call, but the RegionScannerHolder is not closed. Eventually the scanners will 
> be cleaned up by the lease expiration, but this could cause many scanners to 
> leak during periods of high throttling.
> We could close the newly opened scanner before throwing the throttle 
> exception, but I think it's better to not open the scanner at all until we've 
> grabbed some quota.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28001) Add request attribute support to BufferedMutator

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28001.
--
Resolution: Fixed

> Add request attribute support to BufferedMutator
> 
>
> Key: HBASE-28001
> URL: https://issues.apache.org/jira/browse/HBASE-28001
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ray Mattingly
>Assignee: Evelyn Boland
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> In https://issues.apache.org/jira/browse/HBASE-27657 we added support for 
> specifying connection and request attributes. One oversight was including 
> support for doing so via the BufferedMutator class. We should add such 
> support in a follow up PR.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28346) Expose checkQuota to Coprocessor Endpoints

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28346.
--
Resolution: Fixed

> Expose checkQuota to Coprocessor Endpoints
> --
>
> Key: HBASE-28346
> URL: https://issues.apache.org/jira/browse/HBASE-28346
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Charles Connell
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Coprocessor endpoints may do non-trivial amounts of work, yet quotas do not 
> throttle them. We can't generically apply quotas to coprocessors because we 
> have no information on what a particular endpoint might do. One thing we 
> could do is expose checkQuota to the RegionCoprocessorEnvironment. This way, 
> coprocessor authors have the tools to ensure that quotas cover their 
> implementations.
> While adding this, we can update AggregationImplementation to call checkQuota 
> since those endpoints can be quite expensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Proposal to change meta operation timeout default - HBASE-28608

[Nick Dimiduk]

Hi Daniel,

I think that this change makes sense -- an unconfigured client meta request 
timeout should fall-back to the client request timeout -- however it is 
configured.

Thanks,
Nick

On 2024/10/01 20:15:15 "Daniel Roudnitsky (BLOOMBERG/ 919 3RD A)" wrote:
> Hello everyone,
> 
> In HBASE-28608 I proposed a change to the default meta operation timeout, a 
> behavioral change to the client. As its a behavioral change to the client I 
> was suggested to start a dev list thread to get more opinions on the proposal 
> (thank you Duo for suggestion and review). I would appreciate any reviews on 
> this issue - https://issues.apache.org/jira/browse/HBASE-28608
> 
> Thank you all,
> Daniel Roudnitsky

Re: [DISCUSS] Migrate from Jenkins to GitHub Actions for CI

[Nick Dimiduk]

Hi Duo, welcome back.

Sorry I don't understand your point about using Github discussions or
issues. I didn't mean to suggest adoption of those features, only Actions.

I do think that there may be other project management features offered by
Github that could be useful to us, but I'm less clear on what the wins
might be, how they interact with ASF policies about self-hosting, or how we
integrate them into our processes.

On Tue, Oct 8, 2024 at 9:34 AM 张铎(Duo Zhang)  wrote:

> Oh, typo, PMCs -> PMC members
>
> 张铎(Duo Zhang)  于2024年10月8日周二 11:46写道：
> >
> > For me I think we could first enable the github issues, for users to
> > ask questions and discuss things. And if there are actual bugs or
> > something which require code changes, we could file an jira, and also
> > let the contributor to register a jira account.
> > I think this is also easier for our PMCs to decide whether a jira
> > account is necessary for a given user comparing to the current
> > workflow. The private mailing list is full of jira registrations
> > notifications and hard to find other useful information...
> >
> > And on moving to github actions, in general I'm +1 on this. We should
> > try to follow modern ways.
> >
> > And on the funding side, we still have 10 machines, we could contact
> > INFRA to see how to make use of these machines if we switch to github
> > actions.
> >
> > Thanks.
> >
> > Istvan Toth  于2024年10月2日周三 17:31写道：
> > >
> > > I've been working on modifying the existing Jenkinsfile, and it has
> been a
> > > horrible experience, especially as I'm trying to mix declarative and
> > > scripted syntax.
> > > I think from a usability standpoint GH actions would be a win.
> > >
> > > On the other hand, our Jenkinsfiles don't do that much, as most of the
> > > actual CI process is performed via Yetus, so migration shouldn't be a
> huge
> > > amount of work.
> > >
> > > I seem to recall seeing similar discussions on ASF mailing lists, but I
> > > haven't followed them closely.
> > >
> > > Istvan
> > >
> > > Istvan
> > >
> > > On Wed, Oct 2, 2024 at 11:23 AM Nick Dimiduk 
> wrote:
> > >
> > > > Heya,
> > > >
> > > > I'd like to take the community temperature on migrating our build
> infra
> > > > from the ci-hbase.a.o Jenkins instance to something built on GitHub
> > > > Actions. I have several reasons that justify this proposal.
> > > >
> > > > As some of you may know, our community funding has reduced and we
> will no
> > > > longer be able to sustain the current fleet of build infrastructure.
> So,
> > > > one motivation for this proposal is cost-cutting: I think that we'll
> be
> > > > able to operate at lower costs if we can migrate to a
> provisioned-as-needed
> > > > model of consumption.
> > > >
> > > > My second reason is an optimistic appeal to a larger contributor
> base. I
> > > > suspect that if we can modernize our infrastructure then we will
> increase
> > > > the pool of contributors who might be able to participate in this
> area. I
> > > > believe that GH Actions (and systems like it) is more prevalent in
> the
> > > > industry than Jenkins, which means that more people already have
> experience
> > > > with the platform and more people will feel compelled to offer
> support to
> > > > an OSS project that uses the platform as a means of growing their own
> > > > skillset and as a means of bolstering their CVs.
> > > >
> > > > Dove-tailed into reason two is reason three: I believe that there is
> a
> > > > large community of folks who are developing GitHub Actions on its
> > > > marketplace. We would effectively open ourselves up to more
> off-the-shelf
> > > > offerings and those offerings would be in our hands directly. By
> contrast,
> > > > I don't think there's as much development in Jenkins plugins, and the
> > > > process of adding a new plugin to our Jenkins instance requires
> filing an
> > > > INFRA ticket.
> > > >
> > > > These are my motivations. I'm still not clear on what's possible yet
> for
> > > > ASF projects. I have filed an INFRA ticket, requesting whatever is
> > > > necessary for us to start an experiment. Indeed, I believe that
> there are
> > > > some major limitations on the current implementation provided by the
> ASF,
> > > > and as far as I can tell, only one project with a build footprint
> that
> > > > resembles HBase has pursued this effort. I've catalogued the
> applicable
> > > > information that I've found so far on that issue.
> > > >
> > > > https://issues.apache.org/jira/browse/INFRA-26170
> > > >
> > > > Thanks,
> > > > Nick
> > > >
> > >
> > >
> > > --
> > > *István Tóth* | Sr. Staff Software Engineer
> > > *Email*: st...@cloudera.com
> > > cloudera.com <https://www.cloudera.com>
> > > [image: Cloudera] <https://www.cloudera.com/>
> > > [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> > > Cloudera on Facebook] <https://www.facebook.com/cloudera> [image:
> Cloudera
> > > on LinkedIn] <https://www.linkedin.com/company/cloudera>
> > > --
> > > --
>

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I've also generated an Api Compatibility Report that includes
IA.LimitedPrivate for the 2.6.1rc0, for your reference:
https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_2.6.1RC0_Public_LimitedPrivate.html

I hope this will allow us to spot breaking changes without relying on
downstreamers to self-report.

On Fri, Oct 4, 2024 at 3:54 PM Nick Dimiduk  wrote:

> I published a PR [0] with the set of reverts I think need to be applied. I
> generated API compatibility reports for this branch vs. rel/2.6.0, looking
> at just IA.Public [1] and both IA.Public and IA.LimitedPrivate [2]. Please
> bring your comments to the PR.
>
> Thanks,
> Nick
>
> [0]: https://github.com/apache/hbase/pull/6344
> [1]:
> https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_3f8f42be_Public.html
> [2]:
> https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_3f8f42be_Public_LimitedPrivate.html
>
> On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:
>
>> Please vote on this Apache hbase release candidate,
>> hbase-2.6.1RC0
>>
>> The VOTE will remain open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache hbase 2.6.1
>> [ ] -1 Do not release this package because ...
>>
>> The tag to be voted on is 2.6.1RC0:
>>
>>   https://github.com/apache/hbase/tree/2.6.1RC0
>>
>> This tag currently points to git reference
>>
>>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>>
>> The release files, including signatures, digests, as well as CHANGES.md
>> and RELEASENOTES.md included in this RC can be found at:
>>
>>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>>
>> Maven artifacts are available in a staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>>
>> Maven artifacts for hadoop3 are available in a staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>>
>> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>>
>>   https://downloads.apache.org/hbase/KEYS
>>
>> To learn more about Apache hbase, please see
>>
>>   http://hbase.apache.org/
>>
>> Thanks,
>> Your HBase Release Manager
>>
>

[jira] [Reopened] (HBASE-28396) Quota throttling can cause a leak of scanners

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28396?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28396:
--

Reopening while we resolve the branch-2.6 compatibility issue flagged on the 
2.6.1rc0 VOTE thread, 
https://lists.apache.org/thread/j3sv12msdcpk9sh4g7hq5v8q560zknjn

This one has no api incompatibility itself, it only needs an addendum in the 
event that HBASE-28346 is reverted. 

> Quota throttling can cause a leak of scanners
> -
>
> Key: HBASE-28396
> URL: https://issues.apache.org/jira/browse/HBASE-28396
> Project: HBase
>  Issue Type: Bug
>Reporter: Bryan Beaudreault
>Assignee: Hernan Gelaf-Romer
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> In RSRpcServices.scan, we check the quota after having created a new 
> RegionScannerHolder. If the quota is exceeded, an exception will be thrown. 
> In this case, we can't send the scannerName back to the client because it's 
> just an exception. So the client will be forced to retry the openScanner 
> call, but the RegionScannerHolder is not closed. Eventually the scanners will 
> be cleaned up by the lease expiration, but this could cause many scanners to 
> leak during periods of high throttling.
> We could close the newly opened scanner before throwing the throttle 
> exception, but I think it's better to not open the scanner at all until we've 
> grabbed some quota.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28001) Add request attribute support to BufferedMutator

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28001?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28001:
--

Reopening while we resolve the branch-2.6 compatibility issue flagged on the 
2.6.1rc0 VOTE thread, 
https://lists.apache.org/thread/j3sv12msdcpk9sh4g7hq5v8q560zknjn

> Add request attribute support to BufferedMutator
> 
>
> Key: HBASE-28001
> URL: https://issues.apache.org/jira/browse/HBASE-28001
> Project: HBase
>  Issue Type: Improvement
>Reporter: Ray Mattingly
>Assignee: Evelyn Boland
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> In https://issues.apache.org/jira/browse/HBASE-27657 we added support for 
> specifying connection and request attributes. One oversight was including 
> support for doing so via the BufferedMutator class. We should add such 
> support in a follow up PR.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28842) TestRequestAttributes should fail when expected

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28842?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28842:
--

Reopening while we resolve the branch-2.6 compatibility issue flagged on the 
2.6.1rc0 VOTE thread, 
https://lists.apache.org/thread/j3sv12msdcpk9sh4g7hq5v8q560zknjn

> TestRequestAttributes should fail when expected
> ---
>
> Key: HBASE-28842
> URL: https://issues.apache.org/jira/browse/HBASE-28842
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.0.0, 3.0.0
>Reporter: Evelyn Boland
>Assignee: Evelyn Boland
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Problem:
> The tests in the TestRequestAttributes class pass even when they should fail. 
> I've included an example of a test that should fail but does not below.
> Fix:
> Throw an IOException in the AttributesCoprocessor when the map of expected 
> request attributes does not match the map of given request attributes.
>  
> Test: 
> We set 2+ request attributes on the Get request but always return 0 request 
> attributes from AttributesCoprocessor::getRequestAttributesForRowKey method. 
> Yet the test passes even though the map of expected request attributes never 
> matches the map of given request attributes.
> {code:java}
> @Category({ ClientTests.class, MediumTests.class })
> public class TestRequestAttributes {
>   @ClassRule
>   public static final HBaseClassTestRule CLASS_RULE =
> HBaseClassTestRule.forClass(TestRequestAttributes.class);
>   private static final byte[] ROW_KEY1 = Bytes.toBytes("1");
>   private static final Map> 
> ROW_KEY_TO_REQUEST_ATTRIBUTES =
> new HashMap<>();
>   static {
> CONNECTION_ATTRIBUTES.put("clientId", Bytes.toBytes("foo"));
> ROW_KEY_TO_REQUEST_ATTRIBUTES.put(ROW_KEY1, addRandomRequestAttributes());
>   }
>   private static final ExecutorService EXECUTOR_SERVICE = 
> Executors.newFixedThreadPool(100);
>   private static final byte[] FAMILY = Bytes.toBytes("0");
>   private static final TableName TABLE_NAME = 
> TableName.valueOf("testRequestAttributes");
>   private static final HBaseTestingUtil TEST_UTIL = new HBaseTestingUtil();
>   private static SingleProcessHBaseCluster cluster;
>   @BeforeClass
>   public static void setUp() throws Exception {
> cluster = TEST_UTIL.startMiniCluster(1);
> Table table = TEST_UTIL.createTable(TABLE_NAME, new byte[][] { FAMILY }, 
> 1,
>   HConstants.DEFAULT_BLOCKSIZE, AttributesCoprocessor.class.getName());
> table.close();
>   }
>   @AfterClass
>   public static void afterClass() throws Exception {
> cluster.close();
> TEST_UTIL.shutdownMiniCluster();
>   }
> @Test
> public void testRequestAttributesGet() throws IOException {
>   Configuration conf = TEST_UTIL.getConfiguration();
>   try (
> Connection conn = ConnectionFactory.createConnection(conf, null, 
> AuthUtil.loginClient(conf),
>   CONNECTION_ATTRIBUTES);
> Table table = configureRequestAttributes(conn.getTableBuilder(TABLE_NAME, 
> EXECUTOR_SERVICE),
>   ROW_KEY_TO_REQUEST_ATTRIBUTES.get(ROW_KEY1)).build()) {
> table.get(new Get(ROW_KEY1));
>   }
> }
> private static Map addRandomRequestAttributes() {
>   Map requestAttributes = new HashMap<>();
>   int j = Math.max(2, (int) (10 * Math.random()));
>   for (int i = 0; i < j; i++) {
> requestAttributes.put(String.valueOf(i), 
> Bytes.toBytes(UUID.randomUUID().toString()));
>   }
>   return requestAttributes;
> }
> public static class AttributesCoprocessor implements RegionObserver, 
> RegionCoprocessor {
> @Override
> public Optional getRegionObserver() {
>   return Optional.of(this);
> }
> @Override
> public void preGetOp(ObserverContext c, Get 
> get,
>   List result) throws IOException {
>
> validateRequestAttributes(getRequestAttributesForRowKey(get.getRow(;   
> }
> private Map getRequestAttributesForRowKey(byte[] rowKey) {
>   return Collections.emptyMap(); // This line helps demonstrate the bug
> }
> private boolean validateRequestAttributes(Map 
> requestAttributes) {
>   RpcCall rpcCall = RpcServer.getCurrentCall().get();
>   Map attrs = rpcCall.getRequestAttributes();
>   if (attrs.size() != requestAttributes.size()) {
> return;
>   }
>   for (Map.Entry attr : attrs.entrySet()) {
> if (!requestAttributes.containsKey(attr.getKey())) {
>   return;
> }
> if (!Arrays.equals(requestAttributes.get(attr.getKey()), 
> attr.getValue())) {
>   return;
> }
>   }
>   return;
> }
>   }
> } {code}
>  
>  



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28346) Expose checkQuota to Coprocessor Endpoints

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28346?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28346:
--

Reopening while we resolve the branch-2.6 compatibility issue flagged on the 
2.6.1rc0 VOTE thread, 
https://lists.apache.org/thread/j3sv12msdcpk9sh4g7hq5v8q560zknjn

> Expose checkQuota to Coprocessor Endpoints
> --
>
> Key: HBASE-28346
> URL: https://issues.apache.org/jira/browse/HBASE-28346
> Project: HBase
>  Issue Type: Improvement
>Reporter: Bryan Beaudreault
>Assignee: Charles Connell
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Coprocessor endpoints may do non-trivial amounts of work, yet quotas do not 
> throttle them. We can't generically apply quotas to coprocessors because we 
> have no information on what a particular endpoint might do. One thing we 
> could do is expose checkQuota to the RegionCoprocessorEnvironment. This way, 
> coprocessor authors have the tools to ensure that quotas cover their 
> implementations.
> While adding this, we can update AggregationImplementation to call checkQuota 
> since those endpoints can be quite expensive.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I published a PR [0] with the set of reverts I think need to be applied. I
generated API compatibility reports for this branch vs. rel/2.6.0, looking
at just IA.Public [1] and both IA.Public and IA.LimitedPrivate [2]. Please
bring your comments to the PR.

Thanks,
Nick

[0]: https://github.com/apache/hbase/pull/6344
[1]:
https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_3f8f42be_Public.html
[2]:
https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_3f8f42be_Public_LimitedPrivate.html

On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC0:
>
>   https://github.com/apache/hbase/tree/2.6.1RC0
>
> This tag currently points to git reference
>
>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Created] (HBASE-28901) checkcompatibility.py can run maven commands with parallelism

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28901:


 Summary: checkcompatibility.py can run maven commands with 
parallelism
 Key: HBASE-28901
 URL: https://issues.apache.org/jira/browse/HBASE-28901
 Project: HBase
  Issue Type: Task
  Components: create-release
Reporter: Nick Dimiduk
Assignee: Nick Dimiduk


We can speed up the create-release process by taking advantage of maven 
parallelism during creation of the API compatibility report.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Performance regression after HBASE-27788

[Nick Dimiduk]

Hi Bram, Alejandro,

Thanks for your report. I did see your email as it came in, but I have not
taken time to refamiliarize myself with the behavior of seek hints. I’ve
approved your request for a Jira account. Please do file a ticket, link it
back to the original. If you can quantify the degredation, that would help
a lot. Can you include flame graphs of the region server with and without
the offending patch?

Thanks,
Nick

On Tue, 1 Oct 2024 at 14:31, Bram Schuur 
wrote:

> Dear HBASE developers,
>
> We have a custom hbase filter that seeks (SEEK_NEXT_USING_HINT) to a next
> column family (called "cf") based on data in a cell in a prior column
> family (called "bf_slicing").
>
> We upgraded to hbase 2.6.0 from 2.5.8, the change in this ticket
> https://issues.apache.org/jira/browse/HBASE-27788 caused a
> significant performance degradation. When comparing families here, the 'cf'
> family is ordered lower than 'bf_slicing' due to its length, causing the
> first column family ("bf_slicing") to be fully traversed.
>
> The offending code is here:
>
> https://github.com/apache/hbase/pull/5171/files#diff-1ec9654ed8e00f46e11430fc726f8351db59597723efa0bf1e268196f00244c6R54
>
> The original story (HBASE_27788) mentions no seeking should be done outside
> a column family, but our use case seems legitimate in the data model, so we
> think this is a bug, or are we missing something? If this is a bug, I would
> be happy to file a ticket or provide more information.
>
> Thanks in advance,
>
> Alejandro Acevedo and Bram Schuur
>

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I ran a small ITBLL job against rc0 on a cluster of 9 region servers and 3
masters, generating 125m nodes, using slowDeterministic monkey. The job
took 4 hours and verification succeeded.

On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC0:
>
>   https://github.com/apache/hbase/tree/2.6.1RC0
>
> This tag currently points to git reference
>
>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Created] (HBASE-28899) Explore adopting Revapi for compatibility reporting in PRs and RCs

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28899:


 Summary: Explore adopting Revapi for compatibility reporting in 
PRs and RCs
 Key: HBASE-28899
 URL: https://issues.apache.org/jira/browse/HBASE-28899
 Project: HBase
  Issue Type: Task
  Components: build, community, create-release
Reporter: Nick Dimiduk


The first release candidate of 2.6.1 had quite a few incompatible changes. It 
would be great to surface issues at the time when changes are bring introduced, 
instead of at release time. Our current API Compatibility report is quite nice, 
but is driven by a program that carries a restrictive license and so we haven't 
been able to deploy it widely. It happens that I was looking at another ASF 
project this week and noticed that they do have compatibility checking in 
pre-commit, powered by [Revapi|https://revapi.org/], which has the Apache v2 
license.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I think that we also have source-incompatible additions. The new methods
setRequestAttributes on AsyncBufferedMutatorBuilder will result in a
compilation failure for anyone providing their own implementation of that
class. These came with HBASE-28001: Add request attribute support to
BufferedMutator (#6076)

On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC0:
>
>   https://github.com/apache/hbase/tree/2.6.1RC0
>
> This tag currently points to git reference
>
>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

This method came in via HBASE-28346: Expose checkQuota to Coprocessor
Endpoints (#6066).

Our book says:

> LimitedPrivate annotation comes with a set of target consumers for the
interfaces. Those consumers are coprocessors, phoenix, replication endpoint
implementations or similar. At this point, HBase only guarantees source and
binary compatibility for these interfaces between patch versions.

I agree that this is an ABI-breaking change that is protected by our
compatibility statement.

I'm dismayed that this isn't flagged by our compatibility report...

On Thu, Oct 3, 2024 at 10:57 AM Istvan Toth 
wrote:

> I see a similar issue when trying to build Phoenix with it :
>
> [ERROR] Failed to execute goal
> > org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile
> > (default-compile) on project phoenix-core-server: Compilation failure:
> > Compilation failure:
> > [ERROR]
> >
> /home/stoty/workspaces/apache-phoenix/phoenix/phoenix-core-server/src/main/java/org/apache/phoenix/coprocessor/DelegateRegionCoprocessorEnvironment.java:[41,8]
> > org.apache.phoenix.coprocessor.DelegateRegionCoprocessorEnvironment is
> not
> > abstract and does not override abstract method
> > checkBatchQuota(org.apache.hadoop.hbase.regionserver.Region,int,int) in
> > org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
> > [ERROR]
> >
> /home/stoty/workspaces/apache-phoenix/phoenix/phoenix-core-server/src/main/java/org/apache/phoenix/iterate/SnapshotScanner.java:[201,47]
> >  is not abstract
> > and does not override abstract method
> > checkBatchQuota(org.apache.hadoop.hbase.regionserver.Region,int,int) in
> > org.apache.hadoop.hbase.coprocessor.RegionCoprocessorEnvironment
> >
>
> RegionCoprocessorEnvironment is @InterfaceAudience.LimitedPrivate , so this
> should probably also be treated in a similar way.
>
> We certainly can fix this in Phoenix, but only in a new release.
>
> Istvam
>
>
> On Thu, Oct 3, 2024 at 10:21 AM Nick Dimiduk  wrote:
>
> > -1
> >
> > In reviewing the compatibility report, I think that we have a breaking
> > change to the AsyncTable interface [0]. We have introduced a new method
> > without a default implementation. I believe that this is contrary to our
> > Client Binary Compatibility [1] statement:
> >
> > > Client code written to APIs available in a given patch release can run
> > unchanged (no recompilation needed) against the new jars of later patch
> > versions.
> >
> > I believe that this change was introduced via HBASE-28770: Support
> partial
> > results in AggregateImplementation and AsyncAggregationClient (#6167)
> [2].
> >
> > Let's see if we can add a default implementation that preserves binary
> > compatibility.
> >
> > Thanks,
> > Nick
> >
> > [0]:
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_2.6.1RC0.html#Type_Binary_Problems_Medium
> > [1]: https://hbase.apache.org/book.html#hbase.versioning
> > [2]: https://issues.apache.org/jira/browse/HBASE-28770
> >
> > On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk 
> wrote:
> >
> > > Please vote on this Apache hbase release candidate,
> > > hbase-2.6.1RC0
> > >
> > > The VOTE will remain open for at least 72 hours.
> > >
> > > [ ] +1 Release this package as Apache hbase 2.6.1
> > > [ ] -1 Do not release this package because ...
> > >
> > > The tag to be voted on is 2.6.1RC0:
> > >
> > >   https://github.com/apache/hbase/tree/2.6.1RC0
> > >
> > > This tag currently points to git reference
> > >
> > >   ea9ffa81213bfe2d8d764838c7b962c2151624f1
> > >
> > > The release files, including signatures, digests, as well as CHANGES.md
> > > and RELEASENOTES.md included in this RC can be found at:
> > >
> > >   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
> > >
> > > Maven artifacts are available in a staging repository at:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1558/
> > >
> > > Maven artifacts for hadoop3 are available in a staging repository at:
> > >
> > >
> > https://repository.apache.org/content/repositories/orgapachehbase-1559/
> > >
> > > Artifacts were signed with the 0xEF4EBF27 key which can be found in:
> > >
> > >   https://downloads.apache.org/hbase/KEYS
> > >
> > > To learn more about Apache hbase, please see
> > >
> > >   http://hbase.apache.org/
> > >
> > > Thanks,
> > > Your HBase Release Manager
> > >
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: st...@cloudera.com
> cloudera.com <https://www.cloudera.com>
> [image: Cloudera] <https://www.cloudera.com/>
> [image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
> Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
> on LinkedIn] <https://www.linkedin.com/company/cloudera>
> --
> --
>

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

I have posted a proposed addendum for HBASE-28770:
https://github.com/apache/hbase/pull/6339

On Thu, Oct 3, 2024 at 10:20 AM Nick Dimiduk  wrote:

> -1
>
> In reviewing the compatibility report, I think that we have a breaking
> change to the AsyncTable interface [0]. We have introduced a new method
> without a default implementation. I believe that this is contrary to our
> Client Binary Compatibility [1] statement:
>
> > Client code written to APIs available in a given patch release can run
> unchanged (no recompilation needed) against the new jars of later patch
> versions.
>
> I believe that this change was introduced via HBASE-28770: Support partial
> results in AggregateImplementation and AsyncAggregationClient (#6167) [2].
>
> Let's see if we can add a default implementation that preserves binary
> compatibility.
>
> Thanks,
> Nick
>
> [0]:
> https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_2.6.1RC0.html#Type_Binary_Problems_Medium
> [1]: https://hbase.apache.org/book.html#hbase.versioning
> [2]: https://issues.apache.org/jira/browse/HBASE-28770
>
> On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:
>
>> Please vote on this Apache hbase release candidate,
>> hbase-2.6.1RC0
>>
>> The VOTE will remain open for at least 72 hours.
>>
>> [ ] +1 Release this package as Apache hbase 2.6.1
>> [ ] -1 Do not release this package because ...
>>
>> The tag to be voted on is 2.6.1RC0:
>>
>>   https://github.com/apache/hbase/tree/2.6.1RC0
>>
>> This tag currently points to git reference
>>
>>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>>
>> The release files, including signatures, digests, as well as CHANGES.md
>> and RELEASENOTES.md included in this RC can be found at:
>>
>>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>>
>> Maven artifacts are available in a staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>>
>> Maven artifacts for hadoop3 are available in a staging repository at:
>>
>>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>>
>> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>>
>>   https://downloads.apache.org/hbase/KEYS
>>
>> To learn more about Apache hbase, please see
>>
>>   http://hbase.apache.org/
>>
>> Thanks,
>> Your HBase Release Manager
>>
>

[jira] [Reopened] (HBASE-28770) Support partial results in AggregateImplementation and AsyncAggregationClient

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28770:
--

Reopening while we resolve the branch-2.6 compatibility issue flagged on the 
2.6.1rc0 VOTE thread, 
https://lists.apache.org/thread/j3sv12msdcpk9sh4g7hq5v8q560zknjn

> Support partial results in AggregateImplementation and AsyncAggregationClient
> -
>
> Key: HBASE-28770
> URL: https://issues.apache.org/jira/browse/HBASE-28770
> Project: HBase
>  Issue Type: Improvement
>  Components: Client, Coprocessors, Quotas
>Affects Versions: 2.6.0
>Reporter: Charles Connell
>Assignee: Charles Connell
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> Currently there is a gap in the coverage of HBase's quota-based workload 
> throttling. Requests sent by {{[Async]AggregationClient}} reach 
> {{AggregateImplementation}}. This then executes Scans in a way that bypasses 
> the quota system. We see issues with this at Hubspot where clusters suffer 
> under this load and we don't have a good way to protect them.
> In this ticket I'm teaching {{AggregateImplementation}} to optionally stop 
> scanning when a throttle is violated, and send back just the results it has 
> accumulated so far. In addition, it will send back a row key to 
> {{AsyncAggregationClient}}. When the client gets a response with a row key, 
> it will sleep in order to satisfy the throttle, and then send a new request 
> with a scan starting at that row key. This will have the effect of continuing 
> the work where the last request stopped.
> This feature will be unconditionally enabled by {{AsyncAggregationClient}} 
> once this ticket is finished. {{AggregateImplementation}} will not assume 
> that clients support partial results, however, so it can keep supporting 
> older clients. For clients that do not support partial results, throttles 
> will not be respecting, and results will always be complete.
> This feature was [first proposed on the mailing 
> list|https://lists.apache.org/thread/1vqnxb71z7swq2cogz4qg3cn6b10xp4v]. 
> Builds on work in HBASE-28346.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

-1

In reviewing the compatibility report, I think that we have a breaking
change to the AsyncTable interface [0]. We have introduced a new method
without a default implementation. I believe that this is contrary to our
Client Binary Compatibility [1] statement:

> Client code written to APIs available in a given patch release can run
unchanged (no recompilation needed) against the new jars of later patch
versions.

I believe that this change was introduced via HBASE-28770: Support partial
results in AggregateImplementation and AsyncAggregationClient (#6167) [2].

Let's see if we can add a default implementation that preserves binary
compatibility.

Thanks,
Nick

[0]:
https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/api_compare_2.6.0_to_2.6.1RC0.html#Type_Binary_Problems_Medium
[1]: https://hbase.apache.org/book.html#hbase.versioning
[2]: https://issues.apache.org/jira/browse/HBASE-28770

On Tue, Oct 1, 2024 at 10:51 AM Nick Dimiduk  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-2.6.1RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 2.6.1
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 2.6.1RC0:
>
>   https://github.com/apache/hbase/tree/2.6.1RC0
>
> This tag currently points to git reference
>
>   ea9ffa81213bfe2d8d764838c7b962c2151624f1
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1558/
>
> Maven artifacts for hadoop3 are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1559/
>
> Artifacts were signed with the 0xEF4EBF27 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Resolved] (HBASE-28890) RefCnt Leak error when caching index blocks at write time

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28890.
--
Resolution: Fixed

> RefCnt Leak error when caching index blocks at write time
> -
>
> Key: HBASE-28890
> URL: https://issues.apache.org/jira/browse/HBASE-28890
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta-1, 2.7.0, 2.6.1, 2.5.10
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0, 2.7.0, 2.5.11, 2.6.2
>
>
> Following [~bbeaudreault] works from HBASE-27170 that added the (very useful) 
> refcount leak detector, we sometimes see these reports on some branch-2 based 
> deployments:
> {noformat}
> 2024-09-25 10:06:42,413 ERROR 
> org.apache.hbase.thirdparty.io.netty.util.ResourceLeakDetector: LEAK: 
> RefCnt.release() was not called before it's garbage-collected. See 
> https://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:  
> Created at:
> org.apache.hadoop.hbase.nio.RefCnt.(RefCnt.java:59)
> org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
> org.apache.hadoop.hbase.nio.ByteBuff.wrap(ByteBuff.java:550)
> 
> org.apache.hadoop.hbase.io.ByteBuffAllocator.allocate(ByteBuffAllocator.java:357)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.cloneUncompressedBufferWithHeader(HFileBlock.java:1153)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.getBlockForCaching(HFileBlock.java:1215)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.lambda$writeIndexBlocks$0(HFileBlockIndex.java:997)
> java.base/java.util.Optional.ifPresent(Optional.java:178)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:996)
> 
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:635)
> 
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:378)
> 
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:69)
> 
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74)
> 
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:831)
> 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2033)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2878)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2620)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2592)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2462)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:602)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:572)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:65)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:344)
> {noformat}
> It turns out that we always convert the block to a "on-heap" one, inside 
> LruBlockCache.cacheBlock, so when the index block is a SharedMemHFileBlock, 
> the blockForCaching instance in the code 
> [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java#L1076]
>  becomes eligible for GC without releasing buffers/decreasing refcount 
> (leak), right after we return the BlockIndexWriter.writeIndexBlocks call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28890) RefCnt Leak error when caching index blocks at write time

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28890:
--

The patch to branch-2.5 needs {{spotless:apply}}.

> RefCnt Leak error when caching index blocks at write time
> -
>
> Key: HBASE-28890
> URL: https://issues.apache.org/jira/browse/HBASE-28890
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 3.0.0-beta-1, 2.7.0, 2.6.1, 2.5.10
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0, 2.7.0, 2.5.11, 2.6.2
>
>
> Following [~bbeaudreault] works from HBASE-27170 that added the (very useful) 
> refcount leak detector, we sometimes see these reports on some branch-2 based 
> deployments:
> {noformat}
> 2024-09-25 10:06:42,413 ERROR 
> org.apache.hbase.thirdparty.io.netty.util.ResourceLeakDetector: LEAK: 
> RefCnt.release() was not called before it's garbage-collected. See 
> https://netty.io/wiki/reference-counted-objects.html for more information.
> Recent access records:  
> Created at:
> org.apache.hadoop.hbase.nio.RefCnt.(RefCnt.java:59)
> org.apache.hadoop.hbase.nio.RefCnt.create(RefCnt.java:54)
> org.apache.hadoop.hbase.nio.ByteBuff.wrap(ByteBuff.java:550)
> 
> org.apache.hadoop.hbase.io.ByteBuffAllocator.allocate(ByteBuffAllocator.java:357)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.cloneUncompressedBufferWithHeader(HFileBlock.java:1153)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.getBlockForCaching(HFileBlock.java:1215)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.lambda$writeIndexBlocks$0(HFileBlockIndex.java:997)
> java.base/java.util.Optional.ifPresent(Optional.java:178)
> 
> org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeIndexBlocks(HFileBlockIndex.java:996)
> 
> org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:635)
> 
> org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:378)
> 
> org.apache.hadoop.hbase.regionserver.StoreFlusher.finalizeWriter(StoreFlusher.java:69)
> 
> org.apache.hadoop.hbase.regionserver.DefaultStoreFlusher.flushSnapshot(DefaultStoreFlusher.java:74)
> 
> org.apache.hadoop.hbase.regionserver.HStore.flushCache(HStore.java:831)
> 
> org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.flushCache(HStore.java:2033)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushCacheAndCommit(HRegion.java:2878)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2620)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.internalFlushcache(HRegion.java:2592)
> 
> org.apache.hadoop.hbase.regionserver.HRegion.flushcache(HRegion.java:2462)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:602)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.flushRegion(MemStoreFlusher.java:572)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher.access$1000(MemStoreFlusher.java:65)
> 
> org.apache.hadoop.hbase.regionserver.MemStoreFlusher$FlushHandler.run(MemStoreFlusher.java:344)
> {noformat}
> It turns out that we always convert the block to a "on-heap" one, inside 
> LruBlockCache.cacheBlock, so when the index block is a SharedMemHFileBlock, 
> the blockForCaching instance in the code 
> [here|https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlockIndex.java#L1076]
>  becomes eligible for GC without releasing buffers/decreasing refcount 
> (leak), right after we return the BlockIndexWriter.writeIndexBlocks call.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[DISCUSS] Migrate from Jenkins to GitHub Actions for CI

[Nick Dimiduk]

Heya,

I'd like to take the community temperature on migrating our build infra
from the ci-hbase.a.o Jenkins instance to something built on GitHub
Actions. I have several reasons that justify this proposal.

As some of you may know, our community funding has reduced and we will no
longer be able to sustain the current fleet of build infrastructure. So,
one motivation for this proposal is cost-cutting: I think that we'll be
able to operate at lower costs if we can migrate to a provisioned-as-needed
model of consumption.

My second reason is an optimistic appeal to a larger contributor base. I
suspect that if we can modernize our infrastructure then we will increase
the pool of contributors who might be able to participate in this area. I
believe that GH Actions (and systems like it) is more prevalent in the
industry than Jenkins, which means that more people already have experience
with the platform and more people will feel compelled to offer support to
an OSS project that uses the platform as a means of growing their own
skillset and as a means of bolstering their CVs.

Dove-tailed into reason two is reason three: I believe that there is a
large community of folks who are developing GitHub Actions on its
marketplace. We would effectively open ourselves up to more off-the-shelf
offerings and those offerings would be in our hands directly. By contrast,
I don't think there's as much development in Jenkins plugins, and the
process of adding a new plugin to our Jenkins instance requires filing an
INFRA ticket.

These are my motivations. I'm still not clear on what's possible yet for
ASF projects. I have filed an INFRA ticket, requesting whatever is
necessary for us to start an experiment. Indeed, I believe that there are
some major limitations on the current implementation provided by the ASF,
and as far as I can tell, only one project with a build footprint that
resembles HBase has pursued this effort. I've catalogued the applicable
information that I've found so far on that issue.

https://issues.apache.org/jira/browse/INFRA-26170

Thanks,
Nick

[jira] [Created] (HBASE-28895) Bump Avro dependency version

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28895:


 Summary: Bump Avro dependency version
 Key: HBASE-28895
 URL: https://issues.apache.org/jira/browse/HBASE-28895
 Project: HBase
  Issue Type: Task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28892) Update vote template to also propose an email subject line

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28892:


 Summary: Update vote template to also propose an email subject line
 Key: HBASE-28892
 URL: https://issues.apache.org/jira/browse/HBASE-28892
 Project: HBase
  Issue Type: Task
  Components: create-release
Reporter: Nick Dimiduk


The create-release scripts propose an email for the VOTE thread, populated by 
the details of the release candidate. This template should also include a 
proposed subject line for the email.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[VOTE] The first release candidate for Apache HBase 2.6.1 is available

[Nick Dimiduk]

Please vote on this Apache hbase release candidate,
hbase-2.6.1RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 2.6.1
[ ] -1 Do not release this package because ...

The tag to be voted on is 2.6.1RC0:

  https://github.com/apache/hbase/tree/2.6.1RC0

This tag currently points to git reference

  ea9ffa81213bfe2d8d764838c7b962c2151624f1

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/2.6.1RC0/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1558/

Maven artifacts for hadoop3 are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1559/

Artifacts were signed with the 0xEF4EBF27 key which can be found in:

  https://downloads.apache.org/hbase/KEYS

To learn more about Apache hbase, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

[jira] [Created] (HBASE-28891) Resuming a release build from stage "publish-dist" fails due to missing CHANGES.md and RELEASENOTES.md

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28891:


 Summary: Resuming a release build from stage "publish-dist" fails 
due to missing CHANGES.md and RELEASENOTES.md
 Key: HBASE-28891
 URL: https://issues.apache.org/jira/browse/HBASE-28891
 Project: HBase
  Issue Type: Task
  Components: create-release
Reporter: Nick Dimiduk


The do-release scripts seem to be built to support resuming a partial release 
from a "stage". I found that resuming a release at the "publish-dist" stage 
fails due to missing CHANGES.md and RELEASENOTES.md. The script appears to 
assume the files are present in the output directory, which won't be the case 
if the output directory is cleaned after the previous failed attempt. These 
files are populated via the "tag" stage.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28733) Add "2.6 Documentation" to the website

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28733.
--
Resolution: Fixed

Pushed. Thanks a lot [~paksyd]!

> Add "2.6 Documentation" to the website
> --
>
> Key: HBASE-28733
> URL: https://issues.apache.org/jira/browse/HBASE-28733
> Project: HBase
>  Issue Type: Task
>  Components: community, documentation
>Reporter: Nick Dimiduk
>Assignee: Dávid Paksy
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
>
> We have released 2.6 but the website has not been updated with the new API 
> docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Blockers for 2.6.1

[Nick Dimiduk]

Heya team,

I'm landing the last patch flagged for 2.6.1rc0 in the next couple of
hours. Git and Jira are in alignment. Please pause your backports to
branch-2.6 until you see the VOTE thread.

Thanks,
Nick

On Thu, Sep 19, 2024 at 5:25 PM Nick Dimiduk  wrote:

> I've trimmed down the remaining Jiras for 2.6.1 to this list of 5. Several
> of them have already merged to master and are in the process of backports.
> The only one I'm considering a blocker is the backup issue. I believe that
> we could use some test support on 28803 and 28850.
>
> If you have any other fixes that are close and you'd like to see included,
> please @-mention me or reply on this thread.
>
> Next up is to review the nightlies and isolate any egregious failures.
>
> I intend to ITBLL the tip of branch-2.6 in the next couple of days.
>
> With luck, we'll have an RC next week.
>
> Thanks for your support,
> Nick
>
>
> https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%202.6.1%20ORDER%20BY%20key%20ASC
>
> On Wed, Sep 18, 2024 at 10:50 AM Nick Dimiduk  wrote:
>
>> I've filed a tracking ticket for burning down towards 2.6.1. Anyone who
>> would like to participate, please assign a subtask to yourself. If i've
>> forgotten any steps, please add them as well.
>>
>> https://issues.apache.org/jira/browse/HBASE-28847
>>
>> Thanks,
>> Nick
>>
>> On Sun, Sep 15, 2024 at 4:21 AM 张铎(Duo Zhang) 
>> wrote:
>>
>>> Thanks Bryan and Nick.
>>>
>>> I could also help if needed.
>>>
>>> Thanks.
>>>
>>> Nick Dimiduk  于2024年9月15日周日 01:14写道：
>>> >
>>> > I agree — we’re due.
>>> >
>>> > On Sat, 14 Sep 2024 at 17:05, Bryan Beaudreault <
>>> bbeaudrea...@apache.org>
>>> > wrote:
>>> >
>>> > > Sorry for the delay, I've been too busy with other things and expect
>>> that
>>> > > to continue for a while. I’ve spoken to Nick Dimiduk and he’s going
>>> to take
>>> > > over as RM for 2.6. I expect next week he’ll be able to get a sense
>>> of any
>>> > > outstanding blockers and set a date for the release.
>>> > >
>>> > > On Sat, Sep 14, 2024 at 10:47 AM 张铎(Duo Zhang) <
>>> palomino...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > Bump.
>>> > > >
>>> > > > Are we ready for the 2.6.1 release now? We have delayed too much
>>> till
>>> > > > now...
>>> > > >
>>> > > > 张铎(Duo Zhang)  于2024年8月5日周一 22:20写道：
>>> > > > >
>>> > > > > Thanks Bryan!
>>> > > > >
>>> > > > > Bryan Beaudreault  于2024年8月5日周一
>>> 20:59写道：
>>> > > > > >
>>> > > > > > Ray is out on holiday right now, and should be returning next
>>> week.
>>> > > I'm
>>> > > > > > waiting for him and the team to give an update on their
>>> > > backups-related
>>> > > > > > fixes and reviews of Dieter's work. I've been a bit busy with
>>> an
>>> > > > important
>>> > > > > > internal project, so haven't had as much bandwidth myself.
>>> Once those
>>> > > > final
>>> > > > > > issues are merged, I'm happy to do another release
>>> > > > > >
>>> > > > > > On Sat, Jul 27, 2024 at 10:26 AM 张铎(Duo Zhang) <
>>> > > palomino...@gmail.com>
>>> > > > > > wrote:
>>> > > > > >
>>> > > > > > > Are we ready to make the 2.6.1 release now?
>>> > > > > > >
>>> > > > > > > Are there any remaining issues that need to be addressed?
>>> > > > > > >
>>> > > > > > > Thanks.
>>> > > > > > >
>>> > > > > > > Dieter De Paepe  于2024年7月2日周二
>>> > > 22:53写道：
>>> > > > > > > >
>>> > > > > > > > I found 2 more possible causes for data loss when using
>>> multiple
>>> > > > backup
>>> > > > > > > roots. These should also be included in the 2.6.1 in my
>>> opinion.
>

[jira] [Resolved] (HBASE-28879) Bump hbase-thirdparty to 4.1.9

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28879.
--
Resolution: Fixed

> Bump hbase-thirdparty to 4.1.9
> --
>
> Key: HBASE-28879
> URL: https://issues.apache.org/jira/browse/HBASE-28879
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, thirdparty
>Reporter: Duo Zhang
>Assignee: Nick Dimiduk
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28879) Bump hbase-thirdparty to 4.1.9

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28879:
--

I botched the git summaries on the backport commits.

> Bump hbase-thirdparty to 4.1.9
> --
>
> Key: HBASE-28879
> URL: https://issues.apache.org/jira/browse/HBASE-28879
> Project: HBase
>  Issue Type: Task
>  Components: dependencies, thirdparty
>Reporter: Duo Zhang
>Assignee: Nick Dimiduk
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Automatically hiding stale build-bot comments

[Nick Dimiduk]

This seems to be behaving as intended, so I have backported the change to
branch-2.5+.

Thanks,
Nick

On Tue, Sep 17, 2024 at 12:54 PM Nick Dimiduk  wrote:

> Heya!
>
> I've merged a change to our PR build-bot that will hide old stale
> build-bot comments in the PR. Hopefully this will make it easier to have
> discussion between humans on our PRs, while we work through how to migrate
> over to the GitHub Checks API.
>
> This script is a little ham-handed with making API calls. If we get
> rate-limited and it starts causing issues, this is the commit to revert.
> Otherwise, happy reviewing!
>
> Thanks,
> Nick
>
> https://issues.apache.org/jira/browse/HBASE-28642
>

[jira] [Resolved] (HBASE-28642) Hide old PR comments when posting new

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28642.
--
Resolution: Fixed

Pushed to branch-2.5+.

> Hide old PR comments when posting new
> -
>
> Key: HBASE-28642
> URL: https://issues.apache.org/jira/browse/HBASE-28642
> Project: HBase
>  Issue Type: Task
>  Components: build, community
>    Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1, 2.5.11
>
>
> It would be really nice if the build bot would hide the old commits when it 
> posts new ones. When a PR has been open for a while, we end up with more 
> build-bot activity than human activity and it's easy to lose human comments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Purge KeyValue from IA.Public MR classes

[Nick Dimiduk]

Hi Duo,

Sorry for the tardy response.

I think that we need to purge IA.Private classes from public interfaces, so
this is a good project in principle. This should be done for the 3.0
release?

Maybe we should also build an auditing tool that can find these leaking
symbols. Or we can repurpose the API checker implementation for this
purpose?

Thanks,
Nick

On Wed, Jul 3, 2024 at 10:57 AM 张铎(Duo Zhang)  wrote:

> While cleaning up Cell related APIs, I found that there are still
> several classes under hbase-mapreduce use KeyValue in public facing
> APIs, like TextSortReducer, PutSortReducer, etc.
>
> KeyValue is now IA.Private, I think we should change to use Cell instead.
>
> Since KeyValue is a cell, I do not think changing the declaration
> directly will actually break anything.
>
> And we could claim that, before the next major release, the actual
> output type is still KeyValue, although we have changed the
> declaration type. In the next major release, we may change the actual
> output instances to other Cell types so you should change your
> OutputFormat implementation if it requires KeyValue type. If you use
> HFileOutputFormat2, there will be no problem at all.
>
> Thoughts?
>
> Thanks.
>

Re: [DISCUSS] Reverting hbase rest protobuf package name

[Nick Dimiduk]

Hi Istvan,

I think that your assessment is correct and your proposal makes sense to
me. Let's keep it clear that this is a public interface.

For the branch-2 changes, I'm less sure. Well, I agree in principle, but
I'm not sure if this can be achieved without a breaking change. I'm happy
to be proven wrong.

Good on you!

Thanks,
Nick

On Fri, Jul 12, 2024 at 12:42 PM Istvan Toth 
wrote:

> HBASE-23975 was the original ticket.
>
> My guess is that since hbase-shaded-protocol was already set up to do the
> compiling and shading, moving it there was the easiest solution.
> I guess that the same logic was behind the rename: since every other class
> there uses the .shaded. package, change the REST messages the same way.
>
> regards
> Istvan
>
>
>
>
> On Fri, Jul 12, 2024 at 9:48 AM 张铎(Duo Zhang) 
> wrote:
>
> > In which jira we did this moving? Are there any reasons why we did
> > this in the past?
> >
> > Istvan Toth  于2024年7月12日周五 03:57写道：
> > >
> > > Hi!
> > >
> > > While working on HBASE-28725, I realized that in HBase 3+ the REST
> > protobuf
> > > definition files have been moved to hbase-shaded-protobuf, and the
> > package
> > > name has also been renamed.
> > >
> > > While I fully agree with the move to using the thirdparty protobuf
> > library
> > > (in fact I'd like to backport that change to 2.x), I think that moving
> > the
> > > .proto files and renaming the package was not a good idea.
> > >
> > > The REST interface does not use the HBase patched features of the
> > protobuf
> > > library, and if we want to maintain any pretense that the REST protobuf
> > > encoding is usable by non-java code, then we should not use it in the
> > > future either.
> > >
> > > (If we ever decide to use the patched features for performance reasons,
> > we
> > > will need to define new protobuf messages for that anyway)
> > >
> > > Protobuf does not use the package name on the wire, so wire
> compatibility
> > > is not an issue.
> > >
> > > In the unlikely case that someone has implemented an independent REST
> > > client that uses protobuf encoding, this will also ensure compatibility
> > > with the 3.0+ .protoc definitions.
> > >
> > > My proposal is:
> > >
> > > HBASE-28726  Revert
> > REST
> > > protobuf package to org.apache.hadoop.hbase.shaded.rest
> > > *This applies only to branch-3+:*
> > > 1. Move the REST .proto files and compiling back to the hbase-rest
> module
> > > (but use the same protoc compiler that we use now)
> > > 2. Revert the package name of the protobuf messages to the original
> > > 3. No other changes, we still use the thirdparty protobuf library.
> > >
> > > The other issue is that on HBase 2.x the REST client still requires
> > > unshaded protobuf 2.5.0 which brings back all the protobuf library
> > > conflicts that were fixed in 3.0 and by hbase-shaded-client. To fix
> this,
> > > my proposal is:
> > >
> > > HBASE-28725  Use
> > > thirdparty protobuf for REST interface in HBase 2.x
> > > *This applies only to branch-2.x:*
> > > 1. Backport the code changes that use the thirdparty protobuf library
> for
> > > REST to branch-2.x
> > >
> > > With these two changes, the REST code would be almost identical on
> every
> > > branch, easing maintenance.
> > >
> > > What do you think ?
> > >
> > > Istvan
> >
>
>
> --
> *István Tóth* | Sr. Staff Software Engineer
> *Email*: st...@cloudera.com
> cloudera.com 
> [image: Cloudera] 
> [image: Cloudera on Twitter]  [image:
> Cloudera on Facebook]  [image: Cloudera
> on LinkedIn] 
> --
> --
>

Re: Inquiry on Tentative Release Dates for New Patches and Stability Status for HBase 2.6 Branch

[Nick Dimiduk]

Hi Mohammad,

Apologies for my tardy reply -- I don't actively monitor the user list.

We are working now on the 2.6.1 release. You can track progress on this
Jira, https://issues.apache.org/jira/browse/HBASE-28847.

Thanks,
Nick

On Fri, Sep 13, 2024 at 7:21 AM Mohammad Adnan Shaikh 
wrote:

> Hello Hbase Development team,
>
> Hope you all are doing well.
>
> It would be helpful if the team can share some updates/insights for
> requested data.
>
> Thanks,
> Mohammad Adnan Shaikh
>
> On Mon, Sep 9, 2024 at 2:10 PM Mohammad Adnan Shaikh 
> wrote:
>
> > Looping dev team .
> >
> > On Mon, Sep 9, 2024 at 2:08 PM Mohammad Adnan Shaikh  >
> > wrote:
> >
> >> Hello Hbase Community ,
> >>
> >> Hope you are doing well.
> >>
> >> I am writing to inquire about the upcoming release schedule for patches
> >> related to the HBase 2.6 branch. Specifically, I am interested in
> >> understanding the tentative release dates for any new patches that are
> >> planned. I have reviewed the HBase GitHub repository but noticed that
> there
> >> is no tag for version 2.6.1 created yet.
> >>
> >> Additionally, I would like to know which patch version is expected to be
> >> considered as a stable release for the 2.6 branch and its tentative
> release
> >> date.
> >>
> >> Your guidance on these matters would be greatly appreciated, as it will
> >> assist in planning and managing our project's dependencies effectively.
> >>
> >>
> >> Thanks,
> >>
> >> Mohammad Adnan Shaikh
> >>
> >
>

[jira] [Created] (HBASE-28883) Manage hbase-thirdparty transitive dependencies via BOM pom

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28883:


 Summary: Manage hbase-thirdparty transitive dependencies via BOM 
pom
 Key: HBASE-28883
 URL: https://issues.apache.org/jira/browse/HBASE-28883
 Project: HBase
  Issue Type: Task
  Components: build, thirdparty
Reporter: Nick Dimiduk


Despite the intentions to the contrary, there are several places where we need 
the version of a dependency managed in hbase-thirdparty to match an import in 
the main product (and maybe also in our other repos). Right now, this is 
managed via comments in the poms, which read "when this changes there, don't 
for get to update it here...". We can do better than this.

I think that hbase-thirdparty could publish a BOM pom file that can be imported 
into any of the downstream hbase projects that make use of that release of 
hbase-thirdparty. That will centralize management of these dependencies in the 
hbase-thirdparty repo.

This blog post has a nice write-up on the idea, 
https://www.garretwilson.com/blog/2023/06/14/improve-maven-bom-pattern



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Reopened] (HBASE-28642) Hide old PR comments when posting new

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk reopened HBASE-28642:
--

Reopen for backport.

> Hide old PR comments when posting new
> -
>
> Key: HBASE-28642
> URL: https://issues.apache.org/jira/browse/HBASE-28642
> Project: HBase
>  Issue Type: Task
>  Components: build, community
>    Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1
>
>
> It would be really nice if the build bot would hide the old commits when it 
> posts new ones. When a PR has been open for a while, we end up with more 
> build-bot activity than human activity and it's easy to lose human comments.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Attention please on HBASE-28803

[Nick Dimiduk]

Heya,

I'd appreciate some extra eyeballs on HBASE-28803. It addresses a "master
stuck" scenario via abort. It follows the same model as the behavior of the
region server under similar circumstances. I think it's a reasonable
approach to addressing the situation, but we don't currently behave this
way in response to procedure execution issues.

I have a unit test, but I have not been able to reproduce the scenario in a
deployment.

Thanks,
Nick

https://issues.apache.org/jira/browse/HBASE-28803

Re: [VOTE] The first release candiate for hbase-thirdparty 4.1.9 is available

[Nick Dimiduk]

+1

I also posted up PRs to apply this change across the active release lines
(excluding branch-2.5, see my comments on the PRs),
https://github.com/apache/hbase/pull/6295

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_422): ok
 - mvn clean apache-rat:check
* Built from source (1.8.0_422): ok
 - mvn clean install  -DskipTests
* Unit tests pass (1.8.0_422): ok
 - mvn clean package -P runAllTests  -Dsurefire.rerunFailingTestsCount=3


On Mon, Sep 23, 2024 at 5:41 PM Duo Zhang  wrote:

> Please vote on this Apache hbase thirdparty release candidate,
> hbase-thirdparty-4.1.9RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase thirdparty 4.1.9
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 4.1.9RC0:
>
>   https://github.com/apache/hbase-thirdparty/tree/4.1.9RC0
>
> This tag currently points to git reference
>
>   55bba7d64d21590fff22bddc3c072bda5ea656e8
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.1.9RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1556/
>
> Artifacts were signed with the 0x9AD2AE49 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> To learn more about Apache hbase thirdparty, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

Re: [DISCUSS] Blockers for 2.6.1

[Nick Dimiduk]

I've trimmed down the remaining Jiras for 2.6.1 to this list of 5. Several
of them have already merged to master and are in the process of backports.
The only one I'm considering a blocker is the backup issue. I believe that
we could use some test support on 28803 and 28850.

If you have any other fixes that are close and you'd like to see included,
please @-mention me or reply on this thread.

Next up is to review the nightlies and isolate any egregious failures.

I intend to ITBLL the tip of branch-2.6 in the next couple of days.

With luck, we'll have an RC next week.

Thanks for your support,
Nick

https://issues.apache.org/jira/issues/?jql=project%20%3D%20HBASE%20AND%20resolution%20%3D%20Unresolved%20AND%20fixVersion%20%3D%202.6.1%20ORDER%20BY%20key%20ASC

On Wed, Sep 18, 2024 at 10:50 AM Nick Dimiduk  wrote:

> I've filed a tracking ticket for burning down towards 2.6.1. Anyone who
> would like to participate, please assign a subtask to yourself. If i've
> forgotten any steps, please add them as well.
>
> https://issues.apache.org/jira/browse/HBASE-28847
>
> Thanks,
> Nick
>
> On Sun, Sep 15, 2024 at 4:21 AM 张铎(Duo Zhang) 
> wrote:
>
>> Thanks Bryan and Nick.
>>
>> I could also help if needed.
>>
>> Thanks.
>>
>> Nick Dimiduk  于2024年9月15日周日 01:14写道：
>> >
>> > I agree — we’re due.
>> >
>> > On Sat, 14 Sep 2024 at 17:05, Bryan Beaudreault <
>> bbeaudrea...@apache.org>
>> > wrote:
>> >
>> > > Sorry for the delay, I've been too busy with other things and expect
>> that
>> > > to continue for a while. I’ve spoken to Nick Dimiduk and he’s going
>> to take
>> > > over as RM for 2.6. I expect next week he’ll be able to get a sense
>> of any
>> > > outstanding blockers and set a date for the release.
>> > >
>> > > On Sat, Sep 14, 2024 at 10:47 AM 张铎(Duo Zhang) > >
>> > > wrote:
>> > >
>> > > > Bump.
>> > > >
>> > > > Are we ready for the 2.6.1 release now? We have delayed too much
>> till
>> > > > now...
>> > > >
>> > > > 张铎(Duo Zhang)  于2024年8月5日周一 22:20写道：
>> > > > >
>> > > > > Thanks Bryan!
>> > > > >
>> > > > > Bryan Beaudreault  于2024年8月5日周一 20:59写道：
>> > > > > >
>> > > > > > Ray is out on holiday right now, and should be returning next
>> week.
>> > > I'm
>> > > > > > waiting for him and the team to give an update on their
>> > > backups-related
>> > > > > > fixes and reviews of Dieter's work. I've been a bit busy with an
>> > > > important
>> > > > > > internal project, so haven't had as much bandwidth myself. Once
>> those
>> > > > final
>> > > > > > issues are merged, I'm happy to do another release
>> > > > > >
>> > > > > > On Sat, Jul 27, 2024 at 10:26 AM 张铎(Duo Zhang) <
>> > > palomino...@gmail.com>
>> > > > > > wrote:
>> > > > > >
>> > > > > > > Are we ready to make the 2.6.1 release now?
>> > > > > > >
>> > > > > > > Are there any remaining issues that need to be addressed?
>> > > > > > >
>> > > > > > > Thanks.
>> > > > > > >
>> > > > > > > Dieter De Paepe  于2024年7月2日周二
>> > > 22:53写道：
>> > > > > > > >
>> > > > > > > > I found 2 more possible causes for data loss when using
>> multiple
>> > > > backup
>> > > > > > > roots. These should also be included in the 2.6.1 in my
>> opinion.
>> > > > > > > Unfortunately, I only had time to fix one of them:
>> > > > > > > >
>> > > > > > > >
>> > > > > > > >   *   HBASE-28705 (PR available)
>> > > > > > > >
>> > > > > > > >   *   HBASE-28706
>> > > > > > > >
>> > > > > > > > Regards,
>> > > > > > > > Dieter
>> > > > > > > > 
>> > > > > > > > From: Dieter De Paepe 
>> > > > > > > > Sent: 10 June 2024 15:29
>> > > > > > > > To: dev@hb

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

[Nick Dimiduk]

We do it here or we wait for 4.0. I'm inclined toward doing it here
because (i think) we expect coprocessor implementations to necessarily
recompile anyway. Though I admit I've not thought through what features are
enabled by this new generic signature.

On Thu, Sep 19, 2024 at 3:57 PM 张铎(Duo Zhang)  wrote:

> Filed HBASE-28862.
>
> It will break old coprocessor implementations even if they do not use
> any deprecated method. But since it does not change any abilities, I
> think it is OK to include this in a major release as we have already
> done bunch of breaking changes, like changing the protobuf from
> unshaded one to shaded and relocated one...
>
> Thanks.
>
> 张铎(Duo Zhang)  于2024年9月19日周四 17:36写道：
> >
> > Thanks for pointing this TODO out.
> >
> > Let me file an issue abort this. For me I think we should do this in
> 3.0.0.
> >
> > Nick Dimiduk  于2024年9月19日周四 16:46写道：
> > >
> > > Should we take the opportunity to upgrade the generic definitions in
> our
> > > coprocessor signatures as well?
> > >
> > >
> https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101
> > >
> > > On Tue, Sep 17, 2024 at 4:41 PM Duo Zhang  wrote:
> > >
> > > > I've done several rounds of API cleanup in HBASE-24888 and now for me
> > > > I think it is enough for making the new major release.
> > > >
> > > > I've filed HBASE-28844 for aligning the commit history and jira
> > > > issues, it will be a huge project for a major release and may take
> > > > several weeks.
> > > >
> > > > After that, I think we are good to go. And I will also try to find
> > > > some resources to run several rounds of ITBLL and also some upgrading
> > > > tests. Finally, after everything is OK, I will release 3.0.0-beta-2,
> > > > and then cut branch-3.0 from the 3.0.0-beta-2 tag and make the 3.0.0
> > > > release soon.
> > > >
> > > > Hope I can get all these things done within the year 2024.
> > > >
> > > > Thanks.
> > > >
>

Re: [DISCUSS] For 3.0.0-beta-2 and the final 3.0.0 release

[Nick Dimiduk]

Should we take the opportunity to upgrade the generic definitions in our
coprocessor signatures as well?

https://github.com/apache/hbase/blob/edbb145a3f0c6847f444fefdadea5b82d02b0bdb/hbase-server/src/main/java/org/apache/hadoop/hbase/coprocessor/RegionObserver.java#L101

On Tue, Sep 17, 2024 at 4:41 PM Duo Zhang  wrote:

> I've done several rounds of API cleanup in HBASE-24888 and now for me
> I think it is enough for making the new major release.
>
> I've filed HBASE-28844 for aligning the commit history and jira
> issues, it will be a huge project for a major release and may take
> several weeks.
>
> After that, I think we are good to go. And I will also try to find
> some resources to run several rounds of ITBLL and also some upgrading
> tests. Finally, after everything is OK, I will release 3.0.0-beta-2,
> and then cut branch-3.0 from the 3.0.0-beta-2 tag and make the 3.0.0
> release soon.
>
> Hope I can get all these things done within the year 2024.
>
> Thanks.
>

Re: Automatically hiding stale build-bot comments

[Nick Dimiduk]

If anyone has time to push on this further, I've filed a follow-up.

Thanks,
Nick

https://issues.apache.org/jira/browse/HBASE-28859

On Tue, Sep 17, 2024 at 12:54 PM Nick Dimiduk  wrote:

> Heya!
>
> I've merged a change to our PR build-bot that will hide old stale
> build-bot comments in the PR. Hopefully this will make it easier to have
> discussion between humans on our PRs, while we work through how to migrate
> over to the GitHub Checks API.
>
> This script is a little ham-handed with making API calls. If we get
> rate-limited and it starts causing issues, this is the commit to revert.
> Otherwise, happy reviewing!
>
> Thanks,
> Nick
>
> https://issues.apache.org/jira/browse/HBASE-28642
>

[jira] [Created] (HBASE-28859) Migrate to GitHub Checks API for PR Precommit

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28859:


 Summary: Migrate to GitHub Checks API for PR Precommit
 Key: HBASE-28859
 URL: https://issues.apache.org/jira/browse/HBASE-28859
 Project: HBase
  Issue Type: Improvement
  Components: build, community
Reporter: Nick Dimiduk


Our PR pre-commit check is configured to leave comments on the PR with its 
build results. Especially for our 2.x branches, it is leaving upwards of 4 
comments per build. This results in a lot of bot spam, which is distracting 
from human conversations. I've mitigated the issue via HBASE-28642, but it 
would be better if the bot didn't leave comments, but instead used the [Checks 
API|https://docs.github.com/en/rest/checks]. I believe that Yetus is already 
able to do this, and is configured to do so. I think we need to engage with 
INFRA to get our auth token adjusted.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28858) Update downloads.xml

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28858:


 Summary: Update downloads.xml
 Key: HBASE-28858
 URL: https://issues.apache.org/jira/browse/HBASE-28858
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Blockers for 2.6.1

[Nick Dimiduk]

I've filed a tracking ticket for burning down towards 2.6.1. Anyone who
would like to participate, please assign a subtask to yourself. If i've
forgotten any steps, please add them as well.

https://issues.apache.org/jira/browse/HBASE-28847

Thanks,
Nick

On Sun, Sep 15, 2024 at 4:21 AM 张铎(Duo Zhang)  wrote:

> Thanks Bryan and Nick.
>
> I could also help if needed.
>
> Thanks.
>
> Nick Dimiduk  于2024年9月15日周日 01:14写道：
> >
> > I agree — we’re due.
> >
> > On Sat, 14 Sep 2024 at 17:05, Bryan Beaudreault  >
> > wrote:
> >
> > > Sorry for the delay, I've been too busy with other things and expect
> that
> > > to continue for a while. I’ve spoken to Nick Dimiduk and he’s going to
> take
> > > over as RM for 2.6. I expect next week he’ll be able to get a sense of
> any
> > > outstanding blockers and set a date for the release.
> > >
> > > On Sat, Sep 14, 2024 at 10:47 AM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > Bump.
> > > >
> > > > Are we ready for the 2.6.1 release now? We have delayed too much till
> > > > now...
> > > >
> > > > 张铎(Duo Zhang)  于2024年8月5日周一 22:20写道：
> > > > >
> > > > > Thanks Bryan!
> > > > >
> > > > > Bryan Beaudreault  于2024年8月5日周一 20:59写道：
> > > > > >
> > > > > > Ray is out on holiday right now, and should be returning next
> week.
> > > I'm
> > > > > > waiting for him and the team to give an update on their
> > > backups-related
> > > > > > fixes and reviews of Dieter's work. I've been a bit busy with an
> > > > important
> > > > > > internal project, so haven't had as much bandwidth myself. Once
> those
> > > > final
> > > > > > issues are merged, I'm happy to do another release
> > > > > >
> > > > > > On Sat, Jul 27, 2024 at 10:26 AM 张铎(Duo Zhang) <
> > > palomino...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > Are we ready to make the 2.6.1 release now?
> > > > > > >
> > > > > > > Are there any remaining issues that need to be addressed?
> > > > > > >
> > > > > > > Thanks.
> > > > > > >
> > > > > > > Dieter De Paepe  于2024年7月2日周二
> > > 22:53写道：
> > > > > > > >
> > > > > > > > I found 2 more possible causes for data loss when using
> multiple
> > > > backup
> > > > > > > roots. These should also be included in the 2.6.1 in my
> opinion.
> > > > > > > Unfortunately, I only had time to fix one of them:
> > > > > > > >
> > > > > > > >
> > > > > > > >   *   HBASE-28705 (PR available)
> > > > > > > >
> > > > > > > >   *   HBASE-28706
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Dieter
> > > > > > > > 
> > > > > > > > From: Dieter De Paepe 
> > > > > > > > Sent: 10 June 2024 15:29
> > > > > > > > To: dev@hbase.apache.org 
> > > > > > > > Subject: Re: [DISCUSS] Blockers for 2.6.1
> > > > > > > >
> > > > > > > > Hi Bryan,
> > > > > > > >
> > > > > > > > I suggest to add HBASE-28389, and possibly HBASE-28103 to the
> > > list
> > > > as
> > > > > > > well, as they both cause some kind of broken functionality.
> > > > > > > >
> > > > > > > > Regards,
> > > > > > > > Dieter
> > > > > > > > 
> > > > > > > > From: Bryan Beaudreault 
> > > > > > > > Sent: 05 June 2024 21:34
> > > > > > > > To: HBase Dev List 
> > > > > > > > Subject: [DISCUSS] Blockers for 2.6.1
> > > > > > > >
> > > > > > > > Hey all,
> > > > > > > >
> > > > > > > > It's been 2 weeks since 2.6.0 was released. As discussed in
> the
> > > > vote
> > > > > > > > thread, there w

[jira] [Created] (HBASE-28856) Update reporter tool with new release

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28856:


 Summary: Update reporter tool with new release
 Key: HBASE-28856
 URL: https://issues.apache.org/jira/browse/HBASE-28856
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28857) Send announce email

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28857:


 Summary: Send announce email
 Key: HBASE-28857
 URL: https://issues.apache.org/jira/browse/HBASE-28857
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28854) Release version 2.6.1 in Jira

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28854:


 Summary: Release version 2.6.1 in Jira
 Key: HBASE-28854
 URL: https://issues.apache.org/jira/browse/HBASE-28854
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28853) Publish staged repository in Nexus

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28853:


 Summary: Publish staged repository in Nexus
 Key: HBASE-28853
 URL: https://issues.apache.org/jira/browse/HBASE-28853
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28855) Push signed release tag to git

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28855:


 Summary: Push signed release tag to git
 Key: HBASE-28855
 URL: https://issues.apache.org/jira/browse/HBASE-28855
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28852) Propose Release Candidate(s)

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28852:


 Summary: Propose Release Candidate(s)
 Key: HBASE-28852
 URL: https://issues.apache.org/jira/browse/HBASE-28852
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28851) Run a correctness test with ITBLL

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28851:


 Summary: Run a correctness test with ITBLL
 Key: HBASE-28851
 URL: https://issues.apache.org/jira/browse/HBASE-28851
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28849) Review open issues and mark likely inclusion candidates with fixVersion=2.6.1

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28849:


 Summary: Review open issues and mark likely inclusion candidates 
with fixVersion=2.6.1
 Key: HBASE-28849
 URL: https://issues.apache.org/jira/browse/HBASE-28849
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28848) Audit Jira vs. git commit history

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28848:


 Summary: Audit Jira vs. git commit history
 Key: HBASE-28848
 URL: https://issues.apache.org/jira/browse/HBASE-28848
 Project: HBase
  Issue Type: Sub-task
Reporter: Nick Dimiduk






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28847) Release 2.6.1

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28847:


 Summary: Release 2.6.1
 Key: HBASE-28847
 URL: https://issues.apache.org/jira/browse/HBASE-28847
 Project: HBase
  Issue Type: Task
  Components: community
Reporter: Nick Dimiduk


We have 130+ commits on branch-2.6 ; we're over-due for 2.6.1. That's a lot of 
changes, so lets track this a little more carefully than we might a standard 
patch release.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Automatically hiding stale build-bot comments

[Nick Dimiduk]

Heya!

I've merged a change to our PR build-bot that will hide old stale build-bot
comments in the PR. Hopefully this will make it easier to have discussion
between humans on our PRs, while we work through how to migrate over to the
GitHub Checks API.

This script is a little ham-handed with making API calls. If we get
rate-limited and it starts causing issues, this is the commit to revert.
Otherwise, happy reviewing!

Thanks,
Nick

https://issues.apache.org/jira/browse/HBASE-28642

Re: [DISCUSS] Using Hadoop 3.3.6 and 3.4.0 as default versions

[Nick Dimiduk]

t; >
> > > > > When deploying HBase, HBase itself acts as a client of hadoop,
> that's
> > > > > why we always stay on the oldest support hadoop version.
> > > > >
> > > > >
> > > > Not true for 2.6 , which according to the docs supports Hadoop 3.2,
> but
> > > > defaults to Hadoop 3.3
> > > >
> > > >
> > > > > For me, technically I think bumping to the newest patch release of
> a
> > > > > minor release should be fine, which is the proposal 1.
> > > > >
> > > > > But the current hadoopcheck is not enough, since it can only ensure
> > > > > that there is no complation error.
> > > > > Maybe we should also run some simple dev tests in the hadoopcheck
> > > > > stage, and in integration tests, we should try to build with all
> the
> > > > > support hadoop version and run the basic read write tests.
> > > >
> > > >
> > > > Do we need to test all versions ?
> > > > If We test with say, 3.3.0 and 3.3.6 , do we need to test with
> > 3.3.[1-5] ?
> > > > Or if we test with 3.2.5  and 3.3.6, do we need to test with any of
> the
> > > > interim versions ?
> > > >
> > > > Basically, how much do we trust Hadoop to keep to its compatibility
> > rules ?
> > > >
> > > > Running a limited number of tests should not be a problem.
> > > > Should we add a new test category, so that they can be easily started
> > from
> > > > Maven ?
> > > >
> > > > Can you suggest some tests that we should run for the compatibility
> > check ?
> > > >
> > > >
> > > > > Thanks.
> > > > >
> > > > > Istvan Toth  于2024年9月11日周三 21:05写道：
> > > > > >
> > > > > > Let me summarize my take of the discussion so far:
> > > > > >
> > > > > > There are two aspects to the HBase version we build with:
> > > > > > 1. Source code quality/compatibility
> > > > > > 2. Security and usability of the public binary assemblies and
> > (shaded)
> > > > > > hbase maven artifacts.
> > > > > >
> > > > > > 1. Source code quality/compatibility
> > > > > >
> > > > > > AFAICT we have the following hard goals:
> > > > > > 1.a : Ensure that HBase compiles and runs well with the earlier
> > supported
> > > > > > Hadoop version on the given branch
> > > > > > 1.b: Ensure that HBase compiles and runs well with the latest
> > supported
> > > > > > Hadoop version on the given branch
> > > > > >
> > > > > > In my opinion we should also strive for these goals:
> > > > > > 1.c: Aim to officially support the newest possible Hadoop
> releases
> > > > > > 1.d: Take advantage  of new features in newer Hadoop versions
> > > > > >
> > > > > > 2. Public binary usability wish list:
> > > > > >
> > > > > > 2.a: We want them to work OOB for as many use cases as possible
> > > > > > 2.b: We want to work them as well as possible
> > > > > > 2.c: We want to have as few CVEs in them as possible
> > > > > > 2.d: We want to make upgrades as painless as possible, especially
> > for
> > > > > patch
> > > > > > releases
> > > > > >
> > > > > > The factor that Hadoop does not have an explicit end-of-life
> > policy of
> > > > > > course complicates things.
> > > > > >
> > > > > > Our current policy seems to be that we pick a Hadoop version to
> > build
> > > > > with
> > > > > > when releasing a minor version,
> > > > > > and stay on that version until there is a newer patch released of
> > that
> > > > > > minor version with direct CVE fixes.
> > > > > > This does not seem to be an absolute, for example the recently
> > released
> > > > > > HBase 2.4.18 still defaults to Hadoop 3.1.2,
> > > > > > which has several old CVEs, many of which are reportedly fixed in
> > 3.1.3
> > > > > and
> > > > > > 3.1.4.
> > > > > >
> > > > > > my proposals are :
> > >

Re: [DISCUSS] Blockers for 2.6.1

[Nick Dimiduk]

I agree — we’re due.

On Sat, 14 Sep 2024 at 17:05, Bryan Beaudreault 
wrote:

> Sorry for the delay, I've been too busy with other things and expect that
> to continue for a while. I’ve spoken to Nick Dimiduk and he’s going to take
> over as RM for 2.6. I expect next week he’ll be able to get a sense of any
> outstanding blockers and set a date for the release.
>
> On Sat, Sep 14, 2024 at 10:47 AM 张铎(Duo Zhang) 
> wrote:
>
> > Bump.
> >
> > Are we ready for the 2.6.1 release now? We have delayed too much till
> > now...
> >
> > 张铎(Duo Zhang)  于2024年8月5日周一 22:20写道：
> > >
> > > Thanks Bryan!
> > >
> > > Bryan Beaudreault  于2024年8月5日周一 20:59写道：
> > > >
> > > > Ray is out on holiday right now, and should be returning next week.
> I'm
> > > > waiting for him and the team to give an update on their
> backups-related
> > > > fixes and reviews of Dieter's work. I've been a bit busy with an
> > important
> > > > internal project, so haven't had as much bandwidth myself. Once those
> > final
> > > > issues are merged, I'm happy to do another release
> > > >
> > > > On Sat, Jul 27, 2024 at 10:26 AM 张铎(Duo Zhang) <
> palomino...@gmail.com>
> > > > wrote:
> > > >
> > > > > Are we ready to make the 2.6.1 release now?
> > > > >
> > > > > Are there any remaining issues that need to be addressed?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Dieter De Paepe  于2024年7月2日周二
> 22:53写道：
> > > > > >
> > > > > > I found 2 more possible causes for data loss when using multiple
> > backup
> > > > > roots. These should also be included in the 2.6.1 in my opinion.
> > > > > Unfortunately, I only had time to fix one of them:
> > > > > >
> > > > > >
> > > > > >   *   HBASE-28705 (PR available)
> > > > > >
> > > > > >   *   HBASE-28706
> > > > > >
> > > > > > Regards,
> > > > > > Dieter
> > > > > > 
> > > > > > From: Dieter De Paepe 
> > > > > > Sent: 10 June 2024 15:29
> > > > > > To: dev@hbase.apache.org 
> > > > > > Subject: Re: [DISCUSS] Blockers for 2.6.1
> > > > > >
> > > > > > Hi Bryan,
> > > > > >
> > > > > > I suggest to add HBASE-28389, and possibly HBASE-28103 to the
> list
> > as
> > > > > well, as they both cause some kind of broken functionality.
> > > > > >
> > > > > > Regards,
> > > > > > Dieter
> > > > > > 
> > > > > > From: Bryan Beaudreault 
> > > > > > Sent: 05 June 2024 21:34
> > > > > > To: HBase Dev List 
> > > > > > Subject: [DISCUSS] Blockers for 2.6.1
> > > > > >
> > > > > > Hey all,
> > > > > >
> > > > > > It's been 2 weeks since 2.6.0 was released. As discussed in the
> > vote
> > > > > > thread, there were a few outstanding backup-related issues. I
> > believe
> > > > > we've
> > > > > > made some progress on some of those.
> > > > > >
> > > > > > I'd like to start compiling a list of important backup-related
> > fixes to
> > > > > > target for the 2.6.1 release so that we can track progress. Can
> > those of
> > > > > > you who are involved (Ray Mattingly, Nick Dimiduck, Dieter De
> > Paepe &
> > > > > team,
> > > > > > and any others) please list any important jiras here?
> > > > > >
> > > > > > With a list of jiras in hand, we can check to make sure blockers
> &
> > > > > > fixVersions are set and use that to track what we need to drill
> > down
> > > > > before
> > > > > > releasing.
> > > > > >
> > > > > > Here's what I know of so far, let me know if I'm missing
> anything:
> > > > > >
> > > > > > Not yet started:
> > > > > > - HBASE-28084: incremental backups should be forbidden after
> > deleting
> > > > > > backups
> > > > > > - HBASE-28602: Incremental backup fails when WALs move
> > > > > > - HBASE-28462: (similar to ^, but in a different part of the
> code)
> > > > > > - HBASE-28538: BackupHFileCleaner is very expensive
> > > > > >
> > > > > > Patch available:
> > > > > > - HBASE-28539: backup merging does not work when using cloud
> > storage as
> > > > > > filesystem
> > > > > > - HBASE-28562: another possible failure cause for incremental
> > backups +
> > > > > > possibly cause of overly big backup metadata
> > > > > >
> > > > > > Resolved:
> > > > > > - HBASE-28502: backed up tables are not listed correctly in
> backup
> > > > > > metadata, which causes unreliable backup validation
> > > > > > - HBASE-28568: the set of tables included in incremental backups
> > might be
> > > > > > too big
> > > > >
> >
>

Re: [ANNOUNCE] New HBase committer Ray Mattingly

[Nick Dimiduk]

Congratulations Ray and thanks for all the contributions!

On Tue, Sep 10, 2024 at 3:57 AM Viraj Jasani  wrote:

> On behalf of the Apache HBase PMC, I am pleased to announce that
> Ray Mattingly has accepted the PMC's invitation to become a
> committer on the project. We appreciate all of Ray Mattingly's
> generous contributions thus far and look forward to their continued
> involvement.
>
> Congratulations and welcome, Ray Mattingly!
>

Re: [DISCUSS] Using Hadoop 3.3.6 and 3.4.0 as default versions

[Nick Dimiduk]

Yes, we’ll use reflection to make use of APIs introduced in newer HDFS
versions than the stated dependency until the stated dependency finally
catches up.

On Mon, 9 Sep 2024 at 19:55, Wei-Chiu Chuang  wrote:

> Reflection is probably the way to go to ensure maximum compatibility TBH
>
> On Mon, Sep 9, 2024 at 10:40 AM Istvan Toth 
> wrote:
>
> > Stephen Wu has kindly sent me the link for the previous email thread:
> > https://lists.apache.org/thread/2k4tvz3wpg06sgkynkhgvxrodmj86vsj
> >
> > Reading it, I cannot see anything there that would contraindicate
> upgrading
> > to 3.3.6 from 3.3.5, at least on the branches that already default to
> > 3.3.5, i.e. 2.6+.
> >
> > At first glance, the new logic in HBASE-27769 could also be implemented
> > with the usual reflection hacks, while preserving the old logic for
> Hadoop
> > 3.3.5 and earlier.
> >
> > Thanks,
> > Istvan
> >
> >
> >
> > On Mon, Sep 9, 2024 at 1:42 PM Istvan Toth  wrote:
> >
> > > Thanks for your reply, Nick.
> > >
> > > There are no listed direct CVEs in either Hadoop 3.2.4 or 3.3.5, but
> > there
> > > are CVEs in their transitive dependencies.
> > >
> > > My impression is that rather than shipping the oldest 'safe' version,
> > > HBase does seem to update the default Hadoop version to the latest-ish
> at
> > > the time of the start
> > > of the release process, otherwise 2.6 would still default to 3.2.4.
> > (HBase
> > > 2.6 release was already underway when Hadoop 3.4.0 was released)
> > >
> > > For now, we (Phoenix) have resorted to dependency managing transitive
> > > dependencies coming in (only) via Hadoop in Phoenix,
> > > but that is a slippery slope, and adds a layer of uncertainty, as it
> may
> > > introduce incompatibilities in Hadoop that we don't have tests for.
> > >
> > > Our situation is similar to that of the HBase shaded artifacts, where
> we
> > > ship a huge uberjar that includes much of both HBase and Hadoop on top
> of
> > > (or rather below) Phoenix,
> > > similar to the hbase-client-shaded jar.
> > >
> > > I will look into to hadoop check CI tests that you've mentioned, then I
> > > will try to resurrect HBASE-27931, and if I don't find any issues, and
> > > there are no objections, then
> > > I will put a PR to update the unreleased version to default to 3.4.0.
> > >
> > > Istvan
> > >
> > > On Mon, Sep 9, 2024 at 11:06 AM Nick Dimiduk 
> > wrote:
> > >
> > >> My understanding of our hadoop dependency policy is that we ship poms
> > with
> > >> hadoop versions pinned to the oldest compatible, "safe" version that
> is
> > >> supported. Our test infrastructure has a "hadoop check" procedure that
> > >> does
> > >> some validation against other patch release versions.
> > >>
> > >> I don't know if anyone has done a CVE sweep recently. If there are new
> > >> CVEs, we do bump the minimum supported version specified in the pom as
> > >> part
> > >> of patch releases. These changes need to include a pretty thorough
> > >> compatibility check so that we can include release notes about any
> > >> introduced incompatibilities.
> > >>
> > >> I am in favor of a dependency bump so as to address known CVEs as best
> > as
> > >> we reasonably can.
> > >>
> > >> Thanks,
> > >> Nick
> > >>
> > >> On Mon, Sep 9, 2024 at 10:59 AM Istvan Toth  wrote:
> > >>
> > >> > Hi!
> > >> >
> > >> > I'm working on building the Phoenix uberjars with newer Hadoop
> > versions
> > >> by
> > >> > default to improve its CVE stance, and I realized that HBase itself
> > does
> > >> > not use the latest releases.
> > >> >
> > >> > branch-2.5 defaults to 3.2.4
> > >> > branch-2.6 and later defaults to 3.3.5
> > >> >
> > >> > I can kind of understand that we don't want to bump the minor
> version
> > >> for
> > >> > branch-2.5 from the one it was released with.
> > >> >
> > >> > However, I don't see the rationale for not upgrading branch-2.6 to
> at
> > >> least
> > >> > 3.3.6, and the unreleased branches (branch-2, branch-3, master) to

[jira] [Resolved] (HBASE-28643) An unbounded backup failure message can cause an irrecoverable state for the given backup

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28643?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28643.
--
Resolution: Fixed

Pushed to branch-2.6+ . Thanks [~rmdmattingly] !

> An unbounded backup failure message can cause an irrecoverable state for the 
> given backup
> -
>
> Key: HBASE-28643
> URL: https://issues.apache.org/jira/browse/HBASE-28643
> Project: HBase
>  Issue Type: Bug
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> The BackupInfo class has a failedMsg field which is a string of unbounded 
> length. When a DistCp job fails then its failure message contains all of its 
> source paths, and its failure message gets propagated to this failedMsg field 
> on the given BackupInfo.
> If a DistCp job has enough source paths, then this will result in backup 
> status updates being rejected:
> {noformat}
> java.lang.IllegalArgumentException: KeyValue size too large
> at 
> org.apache.hadoop.hbase.client.ConnectionUtils.validatePut(ConnectionUtils.java:513)
> at org.apache.hadoop.hbase.client.HTable.validatePut(HTable.java:1095)
> at org.apache.hadoop.hbase.client.HTable.lambda$put$3(HTable.java:564)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:187)
> at org.apache.hadoop.hbase.client.HTable.put(HTable.java:563)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupSystemTable.updateBackupInfo(BackupSystemTable.java:292)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupManager.updateBackupInfo(BackupManager.java:376)
> at 
> org.apache.hadoop.hbase.backup.impl.TableBackupClient.failBackup(TableBackupClient.java:243)
> at 
> org.apache.hadoop.hbase.backup.impl.IncrementalTableBackupClient.execute(IncrementalTableBackupClient.java:317)
> at 
> org.apache.hadoop.hbase.backup.impl.BackupAdminImpl.backupTables(BackupAdminImpl.java:603)
> at 
> com.hubspot.hbase.recovery.core.backup.BackupManager.lambda$runBackups$2(BackupManager.java:145){noformat}
> Without the ability to update the backup's state, it will never be returned 
> as a failed backup by the client. This means that any mechanisms designed for 
> repairing or cleaning failed backups won't work properly.
> I think that a simple fix here would be fine: we should truncate the 
> failedMsg field to a reasonable maximum size.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] HBase backup API with record/store phase

[Nick Dimiduk]

Hi Dieter,

I don't see a problem with making the individual steps accessible from some
external "driver". My only requirement is that there's a clear interface
between each step so that whatever driver implementations exist don't get
caught with divergent semantics. In the current state, the only driver is
the one that we ship with the project, so there's only one place where such
semantics must be correct. Because this is an area where dataloss is
possible, and dataloss is a reputation-killer for a data storage system
like ours, we must tread carefully.

Thanks,
Nick

On Mon, Jul 15, 2024 at 5:27 PM Dieter De Paepe 
wrote:

> At NGData, we are using HBase backup as part of the backup procedure for
> our product. Besides HBase, some other components (HDFS, ZooKeeper, ...)
> are also backed up.
> Due to how our product works, there are some dependencies between these
> components, i.e. HBase should be backed up first, then ZooKeeper, then...
> To minimize the time between the backup for each component (i.e. to
> minimize data drift), we designed a phased approach in our backup procedure:
>
>   *
> a "record" phase, where all data relevant for a backup is captured. Eg,
> for HDFS this is a HDFS snapshot.
>   *
> a "store" phase, where the captured data is moved to cloud storage. Eg,
> for HDFS, this is a DistCP of that snapshot
>
> This approach allows us to avoid any delay related to data transfer to the
> end of the backup procedure, meaning the time between data capture for all
> component backups is minimized.
>
> The HBase backup API currently doesn't support this kind of phase
> approach, though the steps that are executed certainly would allow this:
>
>   *
> Record phase (full backup): roll WALs, snapshot tables
>   *
> Store phase (full backup): snapshot copy, bulk load copy, updating
> metadata, terminating backup session
>   *
> Record phase (incremental backup): roll WALs
>   *
> Record phase (incremental backup): convert WALs to HFiles, bulk load copy,
> HFile copy, metadata updates, terminating backup session
>
> As this seems like a general use-case, I would like to suggest refactoring
> the HBase backup API to allow this kind of 2-phase approach. CLI usage can
> remain unchanged.
>
> Before logging any ticket about this, I wanted to hear the community's
> thoughts about this.
> Unfortunately, I can't promise we will be available to actually spend time
> on this in the short term, but I'd rather have a plan of attack ready once
> we (or someone else) does have the time.
>
> Regards,
> Dieter
>

[jira] [Created] (HBASE-28733) Publish API docs for 2.6

[Nick Dimiduk (Jira)]

Nick Dimiduk created HBASE-28733:


 Summary: Publish API docs for 2.6
 Key: HBASE-28733
 URL: https://issues.apache.org/jira/browse/HBASE-28733
 Project: HBase
  Issue Type: Task
  Components: community, documentation
Reporter: Nick Dimiduk


We have released 2.6 but the website has not been updated with the new API docs.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28672) Ensure large batches are not indefinitely blocked by quotas

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28672?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-28672.
--
Fix Version/s: 2.7.0
   3.0.0-beta-2
   2.6.1
   Resolution: Fixed

Pushed to branch-2.6+. Thanks [~rmdmattingly] for the contribution and to 
[~zhangduo] for build system quick-fixes.

[~rmdmattingly] should this also go back to 2.5? The patch did not apply 
cleanly, it looked like some interfaces aren't present there. Maybe a 
dependency needs to be backported first?

> Ensure large batches are not indefinitely blocked by quotas
> ---
>
> Key: HBASE-28672
> URL: https://issues.apache.org/jira/browse/HBASE-28672
> Project: HBase
>  Issue Type: Improvement
>  Components: Quotas
>Affects Versions: 2.6.0
>Reporter: Ray Mattingly
>Assignee: Ray Mattingly
>Priority: Major
>  Labels: pull-request-available
> Fix For: 4.0.0-alpha-1, 2.7.0, 3.0.0-beta-2, 2.6.1
>
>
> At my day job we are trying to implement default quotas for a variety of 
> access patterns. We began by introducing a default read IO limit per-user, 
> per-machine — this has been very successful in reducing hotspots, even on 
> clusters with thousands of distinct users.
> While implementing a default writes/second throttle, I realized that doing so 
> would put us in a precarious situation where large-enough batches may never 
> succeed. If your batch size is greater than your TimeLimiter's max 
> throughput, then you will always fail in the quota estimation stage. 
> Meanwhile [IO estimates are more 
> optimistic|https://github.com/apache/hbase/blob/bdb3f216e864e20eb2b09352707a751a5cf7460f/hbase-server/src/main/java/org/apache/hadoop/hbase/quotas/DefaultOperationQuota.java#L192-L193],
>  deliberately, which can let large requests do targeted oversubscription of 
> an IO quota:
>  
> {code:java}
> // assume 1 block required for reads. this is probably a low estimate, which 
> is okay
> readConsumed = numReads > 0 ? blockSizeBytes : 0;{code}
>  
> This is okay because the Limiter's availability will go negative and force a 
> longer backoff on subsequent requests. I believe this is preferable UX 
> compared to a doomed throttling loop.
> In my opinion, we should do something similar in batch request estimation, by 
> estimating a batch request's workload at {{Math.min(batchSize, 
> limiterMaxThroughput)}} rather than simply {{{}batchSize{}}}.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)