date:20211214

Peter Somogyi created HBASE-26570:
-

 Summary: Upgrade to log4j 2.16.0
 Key: HBASE-26570
 URL: https://issues.apache.org/jira/browse/HBASE-26570
 Project: HBase
  Issue Type: Umbrella
  Components: dependencies, hbase-operator-tools
Reporter: Peter Somogyi
Assignee: Peter Somogyi


Log4j just release version 2.16.0 where jndi is turned off by default. Based on 
the release announcement it is not required to fix CVE-2021-44228 but 
recommended. 

[https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26571) [hbase-operator-tools] Upgrade to log4j 2.16.0

Peter Somogyi created HBASE-26571:
-

 Summary: [hbase-operator-tools] Upgrade to log4j 2.16.0
 Key: HBASE-26571
 URL: https://issues.apache.org/jira/browse/HBASE-26571
 Project: HBase
  Issue Type: Sub-task
  Components: dependencies, hbase-operator-tools
Reporter: Peter Somogyi
Assignee: Peter Somogyi






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26572) Upgrade to log4j 2.16.0

Peter Somogyi created HBASE-26572:
-

 Summary: Upgrade to log4j 2.16.0
 Key: HBASE-26572
 URL: https://issues.apache.org/jira/browse/HBASE-26572
 Project: HBase
  Issue Type: Sub-task
  Components: dependencies
Reporter: Peter Somogyi
Assignee: Peter Somogyi
 Fix For: 3.0.0-alpha-2






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26573) [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0

Guangxu Cheng created HBASE-26573:
-

 Summary: [hbase-operator-tools] Generate CHANGES.md and 
RELEASENOTES.md for 1.2.0
 Key: HBASE-26573
 URL: https://issues.apache.org/jira/browse/HBASE-26573
 Project: HBase
  Issue Type: Sub-task
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26574) [hbase-operator-tools] Set version as 1.2.0 in master

Guangxu Cheng created HBASE-26574:
-

 Summary: [hbase-operator-tools] Set version as 1.2.0 in master
 Key: HBASE-26574
 URL: https://issues.apache.org/jira/browse/HBASE-26574
 Project: HBase
  Issue Type: Sub-task
Reporter: Guangxu Cheng
Assignee: Guangxu Cheng






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] First release candidate for hbase 3.0.0-alpha-2 is available for download

We could still add new big features before the last alpha-4 release.

After alpha-4 is out, we will cut a branch-3 and then start to stabilize
it. For branch-3, we should treat it like branch-2. And after beta1 and
beta2, we will cut branch-3.0, to make the final 3.0.0 release.

Thanks.

Mallikarjun  于2021年12月14日周二 10:14写道：

> Not sure if we can add a new feature at this time of the release cycle.
>
> Since native backup/restore will be part of hbase release for the first
> time, these 2 features would be worthwhile to be considered to be part of
> the release.
>
> 1. Add rsgroup support for Backup
> Details: https://issues.apache.org/jira/browse/HBASE-26322
> Patch: https://github.com/apache/hbase/pull/3726
>
> 2. Add support to take parallel backups
> Details: https://issues.apache.org/jira/browse/HBASE-26034
> Patch: https://github.com/apache/hbase/pull/3766
>
>
>
> ---
> Mallikarjun
>
>
> On Tue, Dec 14, 2021 at 5:45 AM 张铎(Duo Zhang) 
> wrote:
>
> > Thanks Josh!
> >
> > Will make a new RC1 soon.
> >
> > Josh Elser  于2021年12月14日周二 04:57写道：
> >
> > > -1 (binding)
> > >
> > > Log4j2 CVE mitigation is ineffective due an incorrect `export` in
> > > bin/hbase-config.sh. Appears that HBASE-26557 tried to add the
> > > mitigation to HBASE_OPTS but added spaces around either side of the
> > > equals sign, e.g. `export HBASE_OPTS = ".."`, which is invalid syntax.
> > >
> > >
> > > 
> > > $ ./bin/start-hbase.sh
> > >
> >
> /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh:
> > >
> > > line 167: export: `=': not a valid identifier
> > >
> >
> /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh:
> > >
> > > line 167: export: ` -Dlog4j2.formatMsgNoLookups=true': not a valid
> > > identifier
> > >
> >
> /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh:
> > >
> > > line 167: export: `=': not a valid identifier
> > >
> >
> /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh:
> > >
> > > line 167: export: ` -Dlog4j2.formatMsgNoLookups=true': not a valid
> > > identifier
> > > 
> > >
> > > More naively, and just in plain bash:
> > > 
> > > bash-5.1$ export FOO = "$FOO bar"
> > > bash: export: `=': not a valid identifier
> > > bash: export: ` bar': not a valid identifier
> > > bash-5.1$ echo $FOO
> > > 
> > >
> > > I'll post a PR to fix after sending this.
> > >
> > > The good:
> > > * xsums and sigs were OK
> > > * Was able to run most unit tests locally
> > > * Was able to launch using bin tarball
> > > * Everything else looks great so far
> > >
> > > - Josh
> > >
> > > On 12/11/21 11:34 AM, Duo Zhang wrote:
> > > > Please vote on this Apache hbase release candidate,
> > > > hbase-3.0.0-alpha-2RC0
> > > >
> > > > The VOTE will remain open for at least 72 hours.
> > > >
> > > > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> > > > [ ] -1 Do not release this package because ...
> > > >
> > > > The tag to be voted on is 3.0.0-alpha-2RC0:
> > > >
> > > >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC0
> > > >
> > > > This tag currently points to git reference
> > > >
> > > >8bca21b47d7c809a0940aea8ed12dd4d2af12432
> > > >
> > > > The release files, including signatures, digests, as well as
> CHANGES.md
> > > > and RELEASENOTES.md included in this RC can be found at:
> > > >
> > > >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC0/
> > > >
> > > > Maven artifacts are available in a staging repository at:
> > > >
> > > >
> > >
> https://repository.apache.org/content/repositories/orgapachehbase-1472/
> > > >
> > > > Artifacts were signed with the 9AD2AE49 key which can be found in:
> > > >
> > > >https://downloads.apache.org/hbase/KEYS
> > > >
> > > > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release
> > > line.
> > > > HBase 3.0.0 includes the following big feature/changes:
> > > >Synchronous Replication
> > > >OpenTelemetry Tracing
> > > >Distributed MOB Compaction
> > > >Backup and Restore
> > > >Move RSGroup balancer to core
> > > >Reimplement sync client on async client
> > > >CPEPs on shaded proto
> > > >Move the logging framework from log4j to log4j2
> > > >
> > > > 3.0.0-alpha-2 contains a critical security fix for addressing the
> > log4j2
> > > > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should
> upgrade
> > > > to 3.0.0-alpha-2 ASAP.
> > > >
> > > > Notice that this is not a production ready release. It is used to let
> > our
> > > > users try and test the new major release, to get feedback before the
> > > final
> > > > GA release is out.
> > > > So please do NOT use it in production. Just try it and report back
> > > > everything you find unusual.
> > > >
> > > > And this time we will not include CHANGES.md and RELEASENOTE.md
> > > > in our source code, you can find it on the download site. For getting
> > > these
> > > > two files for old releases, please go to
> > > >

[VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

Please vote on this Apache hbase release candidate,
hbase-3.0.0-alpha-2RC1

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
[ ] -1 Do not release this package because ...

The tag to be voted on is 3.0.0-alpha-2RC1:

  https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1

This tag currently points to git reference

  a3ff8e4c812eefab6ad32af45ca449a1395a6510

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1473/

Artifacts were signed with the 9AD2AE49 key which can be found in:

  https://downloads.apache.org/hbase/KEYS

3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line.
HBase 3.0.0 includes the following big feature/changes:
  Synchronous Replication
  OpenTelemetry Tracing
  Distributed MOB Compaction
  Backup and Restore
  Move RSGroup balancer to core
  Reimplement sync client on async client
  CPEPs on shaded proto
  Move the logging framework from log4j to log4j2

3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
to 3.0.0-alpha-2 ASAP.

Notice that this is not a production ready release. It is used to let our
users try and test the new major release, to get feedback before the final
GA release is out.
So please do NOT use it in production. Just try it and report back
everything you find unusual.

And this time we will not include CHANGES.md and RELEASENOTE.md
in our source code, you can find it on the download site. For getting these
two files for old releases, please go to

  https://archive.apache.org/dist/hbase/

To learn more about Apache hbase, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

[jira] [Resolved] (HBASE-26554) Introduce a new parameter in jmx servlet to exclude the specific mbean

2021-12-14 Thread Duo Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-26554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26554.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Pushed to master and branch-2.

Thanks [~frostruan] for ocontriburing. Please fill the release note about the 
change here.

> Introduce a new parameter in jmx servlet to exclude the specific mbean
> --
>
> Key: HBASE-26554
> URL: https://issues.apache.org/jira/browse/HBASE-26554
> Project: HBase
>  Issue Type: Improvement
>  Components: metrics
>Affects Versions: 2.4.8
>Reporter: ruanhui
>Assignee: ruanhui
>Priority: Minor
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> There are many regionservers serving over a thousand regions in our company, 
> and the metric load is pretty big.
> I tried to exclude some huge mbean like 
> 'Hadoop:service=HBase,name=RegionServer,sub=Regions' with regex, but didn't 
> succeed.
> So I want to propose a new parameter 'excl' in jmx servlet to exclude the 
> splecific bean or beans.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HBASE-26550) Make sure the master is running normally before accepting a balance command

2021-12-14 Thread Duo Zhang (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-26550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26550.
---
Fix Version/s: 2.5.0
   2.6.0
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.5+.

Thanks [~elserj]!

> Make sure the master is running normally before accepting a balance command
> ---
>
> Key: HBASE-26550
> URL: https://issues.apache.org/jira/browse/HBASE-26550
> Project: HBase
>  Issue Type: Bug
>  Components: Balancer, master
>Reporter: Josh Elser
>Assignee: Josh Elser
>Priority: Minor
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> Noticed this in a unit test from [https://github.com/apache/hbase/pull/3851]
> I believe this is a result of the new balance() implementation in the Master, 
> and a client submitting a request to the master before it's completed its 
> instantiation. Simple fix to avoid the NPE.
> {noformat}
> Failed after attempts=2, exceptions:
> 2021-12-07T18:14:19.203Z, java.io.IOException: java.io.IOException
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:461)
>   at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354)
>   at 
> org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334)
> Caused by: java.lang.NullPointerException
>   at 
> org.apache.hadoop.hbase.master.HMaster.getProcedures(HMaster.java:3335)
>   at 
> org.apache.hadoop.hbase.master.ServerManager.areDeadServersInProgress(ServerManager.java:505)
>   at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1886)
>   at 
> org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:740)
>   at 
> org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java)
>   at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393)
>   ... 3 more {noformat}



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[ANNOUNCE] Apache HBase Thirdparty 4.0.0 released

We've just made a new release of the Apache HBase Thirdparty project.
This project is used by the Apache HBase project to encapsulate a
number of core dependencies that HBase relies upon ensuring
that they are properly isolated from HBase downstream users, e.g.
Google Protocol Buffers, Google Guava, and a few others.

In 4.0.0 we bump the version of some dependencies. In HBASE-25496
we fixed a problem which may cause NoSuchMethodError, please see
HBASE-26492 for more details. And we also shade javax.ws.rs and
jackson-jaxrs-json-provider for use with shaded jersey.

The release is available as source at https://downloads.apache.org/[1]
and as artifacts up in Maven central[2].
Release notes and Changes can be found at [1].

The release was tagged rel/4.0.0.

All artifacts are signed with my key 9AD2AE49 which can be found here:

 https://downloads.apache.org/hbase/KEYS

- Your Release Manager

[1] https://downloads.apache.org/hbase/hbase-thirdparty-4.0.0/
[2]
https://repository.apache.org/service/local/repositories/releases/content/org/apache/hbase/thirdparty/hbase-thirdparty/4.0.0/

Re: [DISCUSS] In flight work to complete before 2.5.0RC0

2021-12-14 Thread Nick Dimiduk

Hi Andrew and team,

The other one I'd like to land for 2.5 is the region visualizer
on HBASE-25865. It stalled for most of the year as it required embedding an
HTTP/JSON API into the Master's WebUI, and I identified issues with our
shading of Jersey and its associated JSR implementations. Since those have
been fixed in hbase-thirdparty-4.0.0, I'd like to revive this feature and
bring it home to conclusion.

There are screenshots of the progress thus far attached to the jira. I've
implemented one visualization, a stacked bar chart that illustrates the
total storefile size per table per region server. As a stretch-goal (or
follow-on), I'd also like to implement a region size histogram. Once we
have a couple in place, there should be enough code infrastructure that
adding other views will be relatively painless. For now, the browser-side
development goes slowly because I have limited JavaScript skills.

As mentioned, this feature relies on an HTTP/JSON endpoint being exposed by
the WebUI that hosts the visualization. I have implemented the bare minimum
required endpoint for the visualization that I have on the
sub-task HBASE-25895. While I intend this to be advertised as an
internal-only, IA.Private API, it deserves some scrutiny.

Thanks,
Nick

On Thu, Dec 9, 2021 at 4:03 PM Nick Dimiduk  wrote:

> The OpenTelemetry improvements are tracked as subtasks of HBASE-26419 [0].
> More eyes on the open PRs would be very helpful, and if anyone is
> interested in joining the discussion in the CNCF community, you can join
> their Slack [1]. One change often depends on the previous, so I have been
> working through the list and pushing patches onto a feature branch [2].
>
> Thanks,
> Nick
>
> [0]: https://issues.apache.org/jira/browse/HBASE-26419
> [1]: https://cloud-native.slack.com/archives/C0150QF88FL
> [2]:
> https://github.com/ndimiduk/hbase/tree/26419-otel-semantic-conventions
>
> On Wed, Dec 8, 2021 at 6:09 PM 张铎(Duo Zhang) 
> wrote:
>
>> OpenTracing -> OpenTelemetry :)
>>
>> For me, I think the OpenTelemetry part is a blocker, we must finish it
>> before cutting an RC since the current implementation is already landed on
>> branch-2.5 and it breaks some Otel best practises, so we should not
>> release
>> it out.
>>
>> Now it is only Nick doing the work and Tak Lon Wu and I reviewing the PRs.
>> And I also joined the CNCF slack channel and saw Nick is working hard in
>> communication with the Otel community on how to better implement tracing
>> in
>> HBase, for example, how to trace big scans.
>> I would encourage more people in our community to involve so we can make
>> progress faster.
>>
>> Thanks.
>>
>> Sean Busbey  于2021年12月9日周四 10:02写道：
>>
>> > If we don't want to wait for HBASE-26543 (fix arg parsing for shell)
>> > then we should revert HBASE-24772 from branch-2.5 prior to an RC.
>> >
>> >
>> > On Wed, Dec 8, 2021 at 7:34 PM Andrew Purtell 
>> wrote:
>> > >
>> > > As your branch-2.5 RM I am assembling a list of work items that
>> should be
>> > > completed before a 2.5.0RC0 candidate is submitted for the PMC's
>> > > consideration.
>> > >
>> > > I have so far:
>> > >
>> > > - OpenTracing span naming convention and coverage improvements.
>> > >
>> > > - Shell exit code fixes/improvements.
>> > >
>> > > - The "encryption improvements umbrella". Arguable, but let's include
>> it
>> > > for now. Can all be resolved as Later if need be.
>> > >
>> > > Let's discuss what else, if anything, should be on this list, or if
>> one
>> > or
>> > > more of the above items does not constitute a release blocker. I
>> consider
>> > > incomplete work-in-progress a blocker. Obviously all of the work in
>> > > progress should land before release. For WIP, let's also agree on a
>> > > definition of done.
>> > >
>> > > --
>> > > Best regards,
>> > > Andrew
>> > >
>> > > Words like orphans lost among the crosstalk, meaning torn from truth's
>> > > decrepit hands
>> > >- A23, Crosstalk
>> >
>>
>

[jira] [Created] (HBASE-26575) StoreHotnessProtector may block Replication

2021-12-14 Thread Bryan Beaudreault (Jira)

Bryan Beaudreault created HBASE-26575:
-

 Summary: StoreHotnessProtector may block Replication
 Key: HBASE-26575
 URL: https://issues.apache.org/jira/browse/HBASE-26575
 Project: HBase
  Issue Type: Bug
Reporter: Bryan Beaudreault


I'm upgrading from hbase1 to hbase2, and I'm still in my QA environment where 
load is very low. Even still, I've noticed some bad interaction between 
Replication and the StoreHotnessProtector.

The ReplicationSink collects edits from the WAL and executes them in batches 
via the normal HTable interface. Despite the name of this property, the max 
batch sizes are based on "hbase.rpc.rows.warning.threshold" which has a default 
of 5000. 

The StoreHotnessProtector defaults to allowing 10 concurrent writes (of 100 
columns or more) to a Store, or 20 concurrent "prepares" of said writes. The 
Prepare part is what causes issues here. When a batch mutate comes in, the RS 
first takes a lock on all rows in the batch. This happens in 
HRegion#lockRowsAndBuildMiniBatch, and the writes are recorded as "preparing" 
in StoreHotnessProtector before acquiring the lock. This recording basically 
increments a counter, and throws an exception if that counter goes over 20.

Back in HRegion#lockRowsAndBuildMiniBatch, the exception is caught and recorded 
in the results for any items that failed. Any items that succeed continue on to 
write,  unless the write is atomic, in which case it immediately throws an 
exception.

This response gets back to the client, which automatically handles retries. 
With enough retries, the batch call will eventually succeed because each retry 
contains fewer and fewer writes to handle. Assuming you have enough retries, 
this is basically enforcing an automatic chunking of of a batch write into 
sub-batches of 20. Again, this only affects writes that hit more than 100 
columns (by default).

At this point I'll say that this in general seems overly aggressive, especially 
since the StoreHotnessProtector doesn't actually do any checks for actual load 
on the RS. You could have a totally idle RegionServer and submit a single batch 
of 100 Puts with 101 columns each – if you don't have at least 5 retries 
configured, the batch will fail.

Back to ReplicationSink, the default batch size is 5000 Puts and the default 
retries is 4. For a table with wide rows (which might cause replication to try 
to sink Puts with more than 100 columns), it becomes basically impossible to 
replicate because the number of retries is not nearly enough to move through a 
batch of up to 5000, 20 at a time.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download

2021-12-14 Thread Nick Dimiduk

+1

* Signature: ok
* Checksum : ok
* Rat check (11.0.11): ok
 - mvn clean apache-rat:check -D skipTests
* Built from source (11.0.11): ok
 - mvn clean install -D skipTests -DskipTests

On Tue, Dec 14, 2021 at 6:16 AM Duo Zhang  wrote:

> Please vote on this Apache hbase release candidate,
> hbase-3.0.0-alpha-2RC1
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 3.0.0-alpha-2RC1:
>
>   https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1
>
> This tag currently points to git reference
>
>   a3ff8e4c812eefab6ad32af45ca449a1395a6510
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1473/
>
> Artifacts were signed with the 9AD2AE49 key which can be found in:
>
>   https://downloads.apache.org/hbase/KEYS
>
> 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line.
> HBase 3.0.0 includes the following big feature/changes:
>   Synchronous Replication
>   OpenTelemetry Tracing
>   Distributed MOB Compaction
>   Backup and Restore
>   Move RSGroup balancer to core
>   Reimplement sync client on async client
>   CPEPs on shaded proto
>   Move the logging framework from log4j to log4j2
>
> 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2
> CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade
> to 3.0.0-alpha-2 ASAP.
>
> Notice that this is not a production ready release. It is used to let our
> users try and test the new major release, to get feedback before the final
> GA release is out.
> So please do NOT use it in production. Just try it and report back
> everything you find unusual.
>
> And this time we will not include CHANGES.md and RELEASENOTE.md
> in our source code, you can find it on the download site. For getting these
> two files for old releases, please go to
>
>   https://archive.apache.org/dist/hbase/
>
> To learn more about Apache hbase, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>

[jira] [Resolved] (HBASE-26570) Upgrade to log4j 2.16.0



 [ 
https://issues.apache.org/jira/browse/HBASE-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Peter Somogyi resolved HBASE-26570.
---
Resolution: Done

> Upgrade to log4j 2.16.0
> ---
>
> Key: HBASE-26570
> URL: https://issues.apache.org/jira/browse/HBASE-26570
> Project: HBase
>  Issue Type: Umbrella
>  Components: dependencies
>Reporter: Peter Somogyi
>Assignee: Peter Somogyi
>Priority: Major
>
> Log4j just release version 2.16.0 where jndi is turned off by default. Based 
> on the release announcement it is not required to fix CVE-2021-44228 but 
> recommended. 
> [https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4]



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26576) Allow Pluggable Queue to belong to FastPath or normal Balanced Executor

2021-12-14 Thread Richard Marscher (Jira)

Richard Marscher created HBASE-26576:


 Summary: Allow Pluggable Queue to belong to FastPath or normal 
Balanced Executor
 Key: HBASE-26576
 URL: https://issues.apache.org/jira/browse/HBASE-26576
 Project: HBase
  Issue Type: New Feature
  Components: regionserver, rpc
Reporter: Richard Marscher
Assignee: Richard Marscher


The PluggableQueue implementation added in[ 
https://github.com/apache/hbase/pull/3522|https://github.com/apache/hbase/pull/3522)]
 for https://issues.apache.org/jira/browse/HBASE-6908 is only wired up with the 
BalancedQueueRpcExecutor. Users should be able to decide their queue operates 
with the FastPathBalancedQueueRpcExecutor optionally via configuration.

I plan to submit a PR for this update soon.

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HBASE-26556) IT and Chaos Monkey improvements

2021-12-14 Thread Wellington Chevreuil (Jira)



 [ 
https://issues.apache.org/jira/browse/HBASE-26556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Wellington Chevreuil resolved HBASE-26556.
--
Resolution: Fixed

Merged into master and cherry-picked into branch-2, branch-2.5, branch-2.4 and 
branch-2.3.

> IT and Chaos Monkey improvements
> 
>
> Key: HBASE-26556
> URL: https://issues.apache.org/jira/browse/HBASE-26556
> Project: HBase
>  Issue Type: Improvement
>  Components: integration tests
>Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.7, 2.4.7, 2.4.8, 2.6.0
>Reporter: Wellington Chevreuil
>Assignee: Wellington Chevreuil
>Priority: Minor
> Fix For: 2.5.0, 2.3.8, 2.4.9, 2.6.0
>
>
> While running the IntegrationTestIngest with slowDeterministic chaos monkey 
> in our environment, we faced some issues with some of the chaos actions that 
> required run some linux shell commands, due to the way we set hbase 
> environment variables and start/stop scripts.
> This patch allows for defining extensions of HBaseShellCommandProvider in the 
> configuration, so that custom commands can be used. It also extends 
> SlowDeterministicMonkeyFactory to provide the ability to configure the 
> destructive actions to be executed.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26577) Update ref guide section for IT and Chaos Monkey to explain the additions from HBASE-26556

2021-12-14 Thread Wellington Chevreuil (Jira)

Wellington Chevreuil created HBASE-26577:


 Summary: Update ref guide section for IT and Chaos Monkey to 
explain the additions from HBASE-26556
 Key: HBASE-26577
 URL: https://issues.apache.org/jira/browse/HBASE-26577
 Project: HBase
  Issue Type: Improvement
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil


HBASE-26556 introduced a customisable monkey factory for the slow deterministic 
policy, as well as made possible for pluggable implementations of the hbase 
remote shell commands. This is to document how these new features can be used.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [DISCUSS] Merge HBASE-26067 branch into master, and backport it to base 2 branches

2021-12-14 Thread Josh Elser

Thanks for your input, Andrew and Nick!

Big thank you to Duo for your hands-on-keyboard commitment as well for
this whole feature.

I am also happy to target 2.x (and not 2.5.x) for the backport.

In the interest of getting rid of this feature branch (and the
inevitable rebase pains the longer it runs parallel to master), I'd like
to move ahead with a concrete plan to merge.

1. Given there was no objection, do folks feel the need for a VOTE? Even
if one person would like a VOTE, I'm happy to start that. Please just
say so.

2. We have three outstanding PRs for the sake of SFT which are all (IMO)
very close to merging (#3851, #3861, and #3942). I think 3851 and 3942
are easy to include and just need one more review cycle. If we feel like
we are still far away on 3861, I think we set that aside and revisit it
after the feature merge is done.

If there are any other concerns, please shout!

- Josh

On 12/8/21 9:07 PM, Andrew Purtell wrote:

+1 for merging to branch-2 (2.6)

On Dec 8, 2021, at 6:04 PM, 张铎 wrote:

I think here we just want this to be backported to 2.x, not 2.5.x.

So thanks Andrew for the quick action.

+1 on merging HBASE-26067 to master and backporting to branch-2(2.6.0).

Thanks.

Andrew Purtell 于2021年12月9日周四 08:45写道：

I concur with Nick, but let me help here by branching 2.5 today. It was
always going to be somewhat arbitrary a point.

On Wed, Dec 8, 2021 at 3:09 PM Nick Dimiduk wrote:

Based solely on the comments made to this thread, I would recommend

against

a merge to branch-2, given that we are very close to 2.5. The points

about

existing gaps seem like things we're not ready to publish in the

impending

minor release. Once we have a branch-2.5, this particular concern of mine
will be alleviated.

Thanks,
Nick

On Wed, Dec 8, 2021 at 1:37 PM Josh Elser wrote:

I was going to wait for some other folks to chime in, but I guess I can
be the next one :)

Duo, Wellington, and Szabolcs have been doing some excellent work on

the

storefile tracking (SFT) to a degree that I never expected to see. I
remember some of the original "Filesystem re-do" issues on Jira. The
idea was exceptional, but the result seemed unreachable.

These devs, building on the success of what Zach/Stephen first talked
about in HBASE-24749, came up with what I think is an excellent step
forward. I've yet to break it via my own testing, but do acknowledge
that there's always more work to be done.

I think this is at a reasonable place to merge this back into the
"mainline" branches from the feature branch (HBASE-26067). I believe
this is ready because:

1. The feature is completely opt-in (HBase works the same way by

default)

2. There is API to migrate tables into the new SFT implementation
3. There is also API to migrate tables back to the default

implementation

Some gaps still exist around bulk loading, documentation, snapshots,

and

recovery tooling, but these are being worked on. In the context of S3,
this makes a significantly more compelling offering of HBase by

removing

the complexity of HBOSS. For HBase in all installations, I think SFT
makes more a significantly more "deterministic" way of managing
regions/files.

+1 from me to merge HBASE-26067 into master and branch-2

- Josh

On 12/7/21 10:31 AM, Wellington Chevreuil wrote:

Hello everyone,

We have been making progress on the alternative way of tracking store

files

originally proposed by Duo in HBASE-26067.

To briefly summarize it for those not following it, this feature

introduces

an abstraction layer to track store files still used/needed by store
engines, allowing for plugging different approaches of identifying

store

files required by the given store. The design doc describing it in

detail is available here
<

https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.calrs3kn4d8s

Our main goal within this feature is to avoid the need for using temp

files

and renames when creating new hfiles (whenever flushing, compacting,
splitting/merging or snapshotting). This is made possible by the

pluggable

tracker implementation labeled "FILE". The current behavior using

temp

dirs

and renames would still be the default approach (labeled "DEFAULT").

This "renameless" approach is appealing for deployments using Amazon

Object store file system, where the lack of atomic rename operations
imposed the necessity of an additional layer of locking (HBOSS),

which

combined with the s3a rename operation can have a performance

overhead.

Some test runs on my employer infrastructure have shown promising

results.

A pure insertion ycsb run has shown ~6% performance gain on the

client

writes. Snapshot clone of hundreds of regions table completes in half

the time. There are also improvements in compaction, splits and

merges

times.

Talking with Duo Zhang and Josh Elser in the HBASE-26067 jira, we

feel

optimistic that the current implementation is in a goo

[jira] [Resolved] (HBASE-26573) [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0



 [ 
https://issues.apache.org/jira/browse/HBASE-26573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guangxu Cheng resolved HBASE-26573.
---
Resolution: Resolved

> [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0
> 
>
> Key: HBASE-26573
> URL: https://issues.apache.org/jira/browse/HBASE-26573
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-14 Thread Guangxu Cheng

Please vote on this Apache hbase operator tools release candidate,
hbase-operator-tools-1.2.0RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase operator tools 1.2.0
[ ] -1 Do not release this package because ...

The tag to be voted on is 1.2.0RC0:

  https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0

This tag currently points to git reference

  76d68624cebb66ec0e509b0a4c0d96445a601685

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:

  https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2
.0RC0/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1474/
Artifacts were signed with the E65E11D40D80DB7C key which can be found in:

  https://dist.apache.org/repos/dist/release/hbase/KEYS

hbase-operator-tools 1.2.0 contains a critical security fix for addressing
the log4j2
CVE-2021-44228. All users who already use hbase-operator-tools 1.1.0 should
upgrade
to hbase-operator-tools 1.2.0 ASAP.

To learn more about Apache hbase operator tools, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-14 Thread Guangxu Cheng

Sorry, some information is missing from the email. Please ignore the email.
--
Best Regards,
Guangxu


Guangxu Cheng  于2021年12月15日周三 11:25写道：

> Please vote on this Apache hbase operator tools release candidate,
> hbase-operator-tools-1.2.0RC0
>
> The VOTE will remain open for at least 72 hours.
>
> [ ] +1 Release this package as Apache hbase operator tools 1.2.0
> [ ] -1 Do not release this package because ...
>
> The tag to be voted on is 1.2.0RC0:
>
>   https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0
>
> This tag currently points to git reference
>
>   76d68624cebb66ec0e509b0a4c0d96445a601685
>
> The release files, including signatures, digests, as well as CHANGES.md
> and RELEASENOTES.md included in this RC can be found at:
>
>   https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2
> .0RC0/
>
> Maven artifacts are available in a staging repository at:
>
>   https://repository.apache.org/content/repositories/orgapachehbase-1474/
> Artifacts were signed with the E65E11D40D80DB7C key which can be found in:
>
>   https://dist.apache.org/repos/dist/release/hbase/KEYS
>
> hbase-operator-tools 1.2.0 contains a critical security fix for addressing
> the log4j2
> CVE-2021-44228. All users who already use hbase-operator-tools 1.1.0
> should upgrade
> to hbase-operator-tools 1.2.0 ASAP.
>
> To learn more about Apache hbase operator tools, please see
>
>   http://hbase.apache.org/
>
> Thanks,
> Your HBase Release Manager
>
>

[VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download

2021-12-14 Thread Guangxu Cheng

Please vote on this Apache hbase operator tools release candidate,
hbase-operator-tools-1.2.0RC0

The VOTE will remain open for at least 72 hours.

[ ] +1 Release this package as Apache hbase operator tools 1.2.0
[ ] -1 Do not release this package because ...

The tag to be voted on is 1.2.0RC0:

  https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0

This tag currently points to git reference

  76d68624cebb66ec0e509b0a4c0d96445a601685

The release files, including signatures, digests, as well as CHANGES.md
and RELEASENOTES.md included in this RC can be found at:


https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/

Maven artifacts are available in a staging repository at:

  https://repository.apache.org/content/repositories/orgapachehbase-1474/
Artifacts were signed with the 5EF3A66D57EC647A key which can be found in:

  https://dist.apache.org/repos/dist/release/hbase/KEYS

hbase-operator-tools 1.2.0 contains a critical security fix for addressing
the log4j2
CVE-2021-44228. All users who use hbase-operator-tools should upgrade
to hbase-operator-tools 1.2.0 ASAP.

To learn more about Apache hbase operator tools, please see

  http://hbase.apache.org/

Thanks,
Your HBase Release Manager

--
Best Regards,
Guangxu

[jira] [Created] (HBASE-26578) CompactedHFilesDischarger should be shutdown when aborting regionserver

2021-12-14 Thread Xiaolin Ha (Jira)

Xiaolin Ha created HBASE-26578:
--

 Summary: CompactedHFilesDischarger should be shutdown when 
aborting regionserver
 Key: HBASE-26578
 URL: https://issues.apache.org/jira/browse/HBASE-26578
 Project: HBase
  Issue Type: Bug
  Components: master, regionserver
Affects Versions: 2.0.0, 3.0.0-alpha-1
Reporter: Xiaolin Ha
Assignee: Xiaolin Ha


The CompactedHFilesDischarger chore service should be shutdown when 
regionserver aborting.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HBASE-26574) [hbase-operator-tools] Set version as 1.2.0 in master



 [ 
https://issues.apache.org/jira/browse/HBASE-26574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guangxu Cheng resolved HBASE-26574.
---
Fix Version/s: hbase-operator-tools-1.2.0
   Resolution: Resolved

> [hbase-operator-tools] Set version as 1.2.0 in master
> -
>
> Key: HBASE-26574
> URL: https://issues.apache.org/jira/browse/HBASE-26574
> Project: HBase
>  Issue Type: Sub-task
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: hbase-operator-tools-1.2.0
>
>




--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download