[jira] [Created] (HBASE-27940) Midkey metadata in root index block would always be ignored by BlockIndexReader.readMultiLevelIndexRoot

[chenglei (Jira)]

chenglei created HBASE-27940:


 Summary: Midkey metadata in root index block would always be 
ignored by BlockIndexReader.readMultiLevelIndexRoot
 Key: HBASE-27940
 URL: https://issues.apache.org/jira/browse/HBASE-27940
 Project: HBase
  Issue Type: Bug
  Components: HFile
Affects Versions: 2.5.5, 2.4.17, 3.0.0-alpha-4, 4.0.0-alpha-1
Reporter: chenglei






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Viraj Jasani]

How about using a new hadoop 3.3 profile for features that are explicitly
present in 3.3 (like FileSystem changes)? When the time comes, we switch to
3.3 profile by default and drop old hadoop 3 profile that supports 3.2.x
versions as of today?


On Fri, Jun 16, 2023 at 7:11 AM 张铎(Duo Zhang)  wrote:

> In general, in HBase, we will use the last patch release of the oldest
> supported hadoop release line as our default hadoop dependency.
>
> For example, since we claim that 3.x will support hadoop 3.2.x and
> 3.3.x, then we will declare the default hadoop version as 3.2.4.
>
> I think we can discuss whether to move up to 3.3.6 as the default
> version, if there are no compatibility issues when communicating with
> 3.2.x hadoop clusters.
>
> But if we want to use the features which are only provided in 3.3.6,
> then we should be careful as this means our users can not build hbase
> with 3.2.x any more, which means we have dropped the support for
> 3.2.x.
>
> Thanks.
>
> Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
> >
> > Hi HBase devs,
> >
> > Over the past few years HBase supports the default Hadoop version of
> 3.2.x
> > but it also works on Hadoop 3.3.x.
> >
> > I'm wondering if it makes sense to move the current default
> hadoop.version
> >  from 3.2.4 to
> > 3.3.x.
> >
> > Why?
> >
> > 1. From a stability and security point of view, Hadoop 3.3 is the most up
> > to date release line. And all HBase tests pass using 3.3.x. There hasn't
> > been a new Hadoop 3.2.x release for over a year.
> >
> > 2. We have a feature (using HBase on Ozone) that depends on an API in
> > Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
> > hadoop version to 3.3.6 will save a lot of hassle.
> >
> > Thoughts?
> >
> > Best,
> > Weichiu
>

Re: Branching for 2.6 code line (branch-2.6)

[Bryan Beaudreault]

In terms of TLS:

- All of our clients (many thousands) in production are using the
NettyRpcConnection with TLS enabled. However, these clients are currently
connecting to the RegionServer/HMaster through an haproxy process local to
each server which handles SSL termination. So not quite end-to-end yet.
- On the server side, most of our QA environment (a thousand regionservers
and ~200 hmasters) are running it. So these are accepting TLS from clients
and using TLS for intra-cluster communication.

The migration is tricky for us due to the scale and the fact that we need
to migrate off haproxy at the same time. Hopefully we should have some of
production running end-to-end TLS within the next month or so.

>From what we've seen in QA so far, there have not been any major issues. We
also couldn't discern any performance issues in testing, though we were
comparing against our legacy haproxy setup and can't really compare against
kerberos.

One outstanding issue is https://issues.apache.org/jira/browse/HBASE-27782,
which we still see periodically. It doesn't seem to cause actual issues,
since the RpcClient still handles it gracefully, but it does cause noise
and may have implications.

On Fri, Jun 16, 2023 at 11:41 AM 张铎(Duo Zhang) 
wrote:

> So any updates here?
>
> Do we have any good news about the TLS usage in production so we can
> move forward on release 2.6.x?
>
> Thanks.
>
> Andrew Purtell  于2023年4月7日周五 09:37写道：
> >
> > Agreed, that sounds like a good plan.
> >
> > On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang) 
> wrote:
> >
> > > I think we could follow the old pattern when we cut a new release
> branch.
> > > That is, after the new release branch is cut and the new minor release
> is
> > > out, we will do a final release of the oldest release line and then
> mark it
> > > as EOL.
> > >
> > > So here, I think once we cut branch-2.6 and release 2.6.0, we can do a
> > > final release for 2.4.x and mark 2.4.x as EOL.
> > >
> > > Thanks.
> > >
> > > Bryan Beaudreault  于2023年3月27日周一 09:57写道：
> > >
> > > > Primary development on hbase-backup and TLS is complete. There are a
> > > couple
> > > > minor things I may want to add to TLS in the future, such as
> pluggable
> > > cert
> > > > verification. But those are not needed for initial release IMO.
> > > >
> > > > We are almost ready integrating hbase-backup in production. We’ve
> fixed a
> > > > few minor things (all committed) but otherwise it’s worked well so
> far in
> > > > tests.
> > > >
> > > > We are a bit delayed in integrating TLS. I’m hopeful it will happen
> in
> > > the
> > > > next 2-3 months. It’s a big project for us, so not quick, but
> definitely
> > > on
> > > > the roadmap.
> > > >
> > > > It seems like cloudera may be closer to integrating TLS in
> production.
> > > > Balazs recently filed and fixed HBASE-27673 related to mTLS. Maybe
> he can
> > > > chime in on his status, or let me know if I am totally off base :)
> > > >
> > > > On Sun, Mar 26, 2023 at 9:25 PM Andrew Purtell <
> andrew.purt...@gmail.com
> > > >
> > > > wrote:
> > > >
> > > > > Before we open a new code line should we discuss EOL of 2.4? After
> the
> > > > > first 2.6 release? It’s not required of course but cuts down the
> amount
> > > > of
> > > > > labor to have two 2.x code lines (presumably, one as stable and
> one as
> > > > > next) rather than three. Perhaps even before that, should we move
> the
> > > > > stable pointer to the latest 2.5 release?
> > > > >
> > > > > >
> > > > > > On Mar 26, 2023, at 5:59 PM, 张铎  wrote:
> > > > > >
> > > > > > Bump.
> > > > > >
> > > > > > I believe the mTLS and backup related code have all been
> finished on
> > > > > > branch-2?
> > > > > >
> > > > > > Are there any other things which block us making the branch-2.6
> > > branch?
> > > > > >
> > > > > > Thanks.
> > > > > >
> > > > > > Mallikarjun  于2022年10月17日周一 02:09写道：
> > > > > >
> > > > > >> On hbase-backup, we are using in production for more then 1
> year. I
> > > > can
> > > > > >> vouch for it to be stable enough to be in a release version so
> that
> > > > more
> > > > > >> people can use it and polished it further.
> > > > > >>
> > > > > >>> On Sun, Oct 16, 2022, 11:25 PM Andrew Purtell <
> > > > > andrew.purt...@gmail.com>
> > > > > >>> wrote:
> > > > > >>>
> > > > > >>> My understanding is some folks evaluating and polishing TLS for
> > > their
> > > > > >>> production are also considering hbase-backup in the same way,
> which
> > > > is
> > > > > >> why
> > > > > >>> I linked them together. If that is incorrect then they both are
> > > still
> > > > > >> worth
> > > > > >>> considering in my opinion but would have a more tenuous link.
> > > > > >>>
> > > > > >>> Where we are with hbase-backup is it should probably be ported
> to
> > > > where
> > > > > >>> more people would be inclined to evaluate it, in order for it
> to
> > > make
> > > > > >> more
> > > > > >>> progress. A new minor releasing line would fit. On the other
> hand
> > > if
> > > > it
> > > > > >> 

[jira] [Resolved] (HBASE-27939) Bump snappy-java from 1.1.9.1 to 1.1.10.1

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27939?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-27939.
---
Fix Version/s: 2.6.0
   2.5.6
   3.0.0-beta-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.5+.

> Bump snappy-java from 1.1.9.1 to 1.1.10.1
> -
>
> Key: HBASE-27939
> URL: https://issues.apache.org/jira/browse/HBASE-27939
> Project: HBase
>  Issue Type: Improvement
>  Components: dependabot, security
>Reporter: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 2.5.6, 3.0.0-beta-1
>
>




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27939) Bump snappy-java from 1.1.9.1 to 1.1.10.1

[Duo Zhang (Jira)]

Duo Zhang created HBASE-27939:
-

 Summary: Bump snappy-java from 1.1.9.1 to 1.1.10.1
 Key: HBASE-27939
 URL: https://issues.apache.org/jira/browse/HBASE-27939
 Project: HBase
  Issue Type: Improvement
  Components: dependabot, security
Reporter: Duo Zhang






--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: Branching for 2.6 code line (branch-2.6)

[Duo Zhang]

So any updates here?

Do we have any good news about the TLS usage in production so we can
move forward on release 2.6.x?

Thanks.

Andrew Purtell  于2023年4月7日周五 09:37写道：
>
> Agreed, that sounds like a good plan.
>
> On Wed, Mar 29, 2023 at 7:31 AM 张铎(Duo Zhang)  wrote:
>
> > I think we could follow the old pattern when we cut a new release branch.
> > That is, after the new release branch is cut and the new minor release is
> > out, we will do a final release of the oldest release line and then mark it
> > as EOL.
> >
> > So here, I think once we cut branch-2.6 and release 2.6.0, we can do a
> > final release for 2.4.x and mark 2.4.x as EOL.
> >
> > Thanks.
> >
> > Bryan Beaudreault  于2023年3月27日周一 09:57写道：
> >
> > > Primary development on hbase-backup and TLS is complete. There are a
> > couple
> > > minor things I may want to add to TLS in the future, such as pluggable
> > cert
> > > verification. But those are not needed for initial release IMO.
> > >
> > > We are almost ready integrating hbase-backup in production. We’ve fixed a
> > > few minor things (all committed) but otherwise it’s worked well so far in
> > > tests.
> > >
> > > We are a bit delayed in integrating TLS. I’m hopeful it will happen in
> > the
> > > next 2-3 months. It’s a big project for us, so not quick, but definitely
> > on
> > > the roadmap.
> > >
> > > It seems like cloudera may be closer to integrating TLS in production.
> > > Balazs recently filed and fixed HBASE-27673 related to mTLS. Maybe he can
> > > chime in on his status, or let me know if I am totally off base :)
> > >
> > > On Sun, Mar 26, 2023 at 9:25 PM Andrew Purtell  > >
> > > wrote:
> > >
> > > > Before we open a new code line should we discuss EOL of 2.4? After the
> > > > first 2.6 release? It’s not required of course but cuts down the amount
> > > of
> > > > labor to have two 2.x code lines (presumably, one as stable and one as
> > > > next) rather than three. Perhaps even before that, should we move the
> > > > stable pointer to the latest 2.5 release?
> > > >
> > > > >
> > > > > On Mar 26, 2023, at 5:59 PM, 张铎  wrote:
> > > > >
> > > > > Bump.
> > > > >
> > > > > I believe the mTLS and backup related code have all been finished on
> > > > > branch-2?
> > > > >
> > > > > Are there any other things which block us making the branch-2.6
> > branch?
> > > > >
> > > > > Thanks.
> > > > >
> > > > > Mallikarjun  于2022年10月17日周一 02:09写道：
> > > > >
> > > > >> On hbase-backup, we are using in production for more then 1 year. I
> > > can
> > > > >> vouch for it to be stable enough to be in a release version so that
> > > more
> > > > >> people can use it and polished it further.
> > > > >>
> > > > >>> On Sun, Oct 16, 2022, 11:25 PM Andrew Purtell <
> > > > andrew.purt...@gmail.com>
> > > > >>> wrote:
> > > > >>>
> > > > >>> My understanding is some folks evaluating and polishing TLS for
> > their
> > > > >>> production are also considering hbase-backup in the same way, which
> > > is
> > > > >> why
> > > > >>> I linked them together. If that is incorrect then they both are
> > still
> > > > >> worth
> > > > >>> considering in my opinion but would have a more tenuous link.
> > > > >>>
> > > > >>> Where we are with hbase-backup is it should probably be ported to
> > > where
> > > > >>> more people would be inclined to evaluate it, in order for it to
> > make
> > > > >> more
> > > > >>> progress. A new minor releasing line would fit. On the other hand
> > if
> > > it
> > > > >> is
> > > > >>> too unpolished then the experience would be poor.
> > > > >>>
> > > > >>>
> > > >  On Oct 16, 2022, at 5:35 AM, 张铎  wrote:
> > > > 
> > > >  I believe the second one is still ongoing?
> > > > 
> > > >  Andrew Purtell  于2022年10月14日周五 05:37写道：
> > > > >
> > > > > We will begin releasing activity for the 2.6 code line and as a
> > > > > prerequisite to that we shall need to make a new branch
> > branch-2.6
> > > > >> from
> > > > > branch-2.
> > > > >
> > > > > Before we do that let's make sure all commits for the key
> > features
> > > of
> > > > >>> 2.6
> > > > > are settled in branch-2 before the branching point. Those key
> > > > features
> > > > >>> are:
> > > > > - mTLS RPC
> > > > > - hbase-backup backport
> > > > >
> > > > > --
> > > > > Best regards,
> > > > > Andrew
> > > > >>>
> > > > >>
> > > >
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
>- A23, Welcome, Apocalypse

[jira] [Resolved] (HBASE-27888) Record readBlock message in log when it takes too long time

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-27888.
---
Fix Version/s: 2.6.0
   2.5.6
   3.0.0-beta-1
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.5+.

The code on branch-2.4 is a bit different so I did not port it to branch-2.4.

Please fill the release note as we added a new config here. [~chaijunjie]

Thanks.

> Record readBlock message in log when it takes too long time
> ---
>
> Key: HBASE-27888
> URL: https://issues.apache.org/jira/browse/HBASE-27888
> Project: HBase
>  Issue Type: Improvement
>  Components: HFile
>Affects Versions: 2.5.3
>Reporter: chaijunjie
>Assignee: chaijunjie
>Priority: Minor
> Fix For: 2.6.0, 2.5.6, 3.0.0-beta-1
>
>
> After HBASE-15160， we can record the readBlock message by TRACE log in 
> org.apache.hadoop.hbase.io.hfile.HFileBlock.FSReaderImpl#readBlockDataInternal，
>  But it record all read block message in TRACE log， some times， we only focus 
> the block read cost too much time..



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Move default Hadoop version to 3.3.x

[Duo Zhang]

In general, in HBase, we will use the last patch release of the oldest
supported hadoop release line as our default hadoop dependency.

For example, since we claim that 3.x will support hadoop 3.2.x and
3.3.x, then we will declare the default hadoop version as 3.2.4.

I think we can discuss whether to move up to 3.3.6 as the default
version, if there are no compatibility issues when communicating with
3.2.x hadoop clusters.

But if we want to use the features which are only provided in 3.3.6,
then we should be careful as this means our users can not build hbase
with 3.2.x any more, which means we have dropped the support for
3.2.x.

Thanks.

Wei-Chiu Chuang  于2023年6月16日周五 06:03写道：
>
> Hi HBase devs,
>
> Over the past few years HBase supports the default Hadoop version of 3.2.x
> but it also works on Hadoop 3.3.x.
>
> I'm wondering if it makes sense to move the current default hadoop.version
>  from 3.2.4 to
> 3.3.x.
>
> Why?
>
> 1. From a stability and security point of view, Hadoop 3.3 is the most up
> to date release line. And all HBase tests pass using 3.3.x. There hasn't
> been a new Hadoop 3.2.x release for over a year.
>
> 2. We have a feature (using HBase on Ozone) that depends on an API in
> Hadoop 3.3.6 that is not yet in any 3.2 release line. Moving the default
> hadoop version to 3.3.6 will save a lot of hassle.
>
> Thoughts?
>
> Best,
> Weichiu

[jira] [Created] (HBASE-27938) Enable PE to load any custom implementation of tests at runtime

[Prathyusha (Jira)]

Prathyusha created HBASE-27938:
--

 Summary: Enable PE to load any custom implementation of tests at 
runtime
 Key: HBASE-27938
 URL: https://issues.apache.org/jira/browse/HBASE-27938
 Project: HBase
  Issue Type: Improvement
  Components: test
Reporter: Prathyusha


Right now to add any custom PE.Test implementation it has to have a compile 
time dependency of those new test classes in PE, this is to enable PE to load 
any custom impl of tests at runtime and utilise PE framework for any custom 
implementations.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

TAC Applications for Community Over Code North America and Asia now open

[Gavin McDonald]

Hi All,

(This email goes out to all our user and dev project mailing lists, so you
may receive this
email more than once.)

The Travel Assistance Committee has opened up applications to help get
people to the following events:


*Community Over Code Asia 2023 - *
*August 18th to August 20th in Beijing , China*

Applications for this event closes on the 6th July so time is short, please
apply as soon as possible. TAC is prioritising applications from the Asia
and Oceania regions.

More details on this event can be found at:
https://apachecon.com/acasia2023/

More information on how to apply please read: https://tac.apache.org/


*Community Over Code North America - *
*October 7th to October 10th in Halifax, Canada*

Applications for this event closes on the 22nd July. We expect many
applications so please do apply as soon as you can. TAC is prioritising
applications from the North and South America regions.

More details on this event can be found at: https://communityovercode.org/

More information on how to apply please read: https://tac.apache.org/


*Have you applied to be a Speaker?*

If you have applied or intend to apply as a Speaker at either of these
events, and think you
may require assistance for Travel and/or Accommodation - TAC advises that
you do not
wait until you have been notified of your speaker status and to apply
early. Should you
not be accepted as a speaker and still wish to attend you can amend you
application to
include Conference fees, or, you may withdraw your application.

The call for presentations for Halifax is here:
https://communityovercode.org/call-for-presentations/
and you have until the 13th of July to apply.

The call for presentations for Beijing is here:
https://apachecon.com/acasia2023/cfp.html
and you have until the 18th June to apply.

*IMPORTANT Note on Visas:*

It is important that you apply for a Visa as soon as possible - do not wait
until you know if you have been accepted for Travel Assistance or not, as
due to current wait times for Interviews in some Countries, waiting that
long may be too late, so please do apply for a Visa right away. Contact
tac-ap...@tac.apache.org if you need any more information or assistance in
this area.

*Spread the Word!!*

TAC encourages you to spread the word about Travel Assistance to get to
these events, so feel free to repost as you see fit on Social Media, at
work, schools, universities etc etc...

Thank You and hope to see you all soon

Gavin McDonald on behalf of the ASF Travel Assistance Committee.

[jira] [Resolved] (HBASE-27935) Introduce Jenkins PR job for hbase-kustomize

[Nick Dimiduk (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27935?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Nick Dimiduk resolved HBASE-27935.
--
Resolution: Fixed

> Introduce Jenkins PR job for hbase-kustomize
> 
>
> Key: HBASE-27935
> URL: https://issues.apache.org/jira/browse/HBASE-27935
> Project: HBase
>  Issue Type: Task
>  Components: build
>Reporter: Nick Dimiduk
>Assignee: Nick Dimiduk
>Priority: Major
>
> We need something to build off of. Let's start with a clone of what's on 
> hbase-operator-tools.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)