[jira] [Created] (HBASE-26570) Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26570: - Summary: Upgrade to log4j 2.16.0 Key: HBASE-26570 URL: https://issues.apache.org/jira/browse/HBASE-26570 Project: HBase Issue Type: Umbrella Components: dependencies, hbase-operator-tools Reporter: Peter Somogyi Assignee: Peter Somogyi Log4j just release version 2.16.0 where jndi is turned off by default. Based on the release announcement it is not required to fix CVE-2021-44228 but recommended. [https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26571) [hbase-operator-tools] Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26571: - Summary: [hbase-operator-tools] Upgrade to log4j 2.16.0 Key: HBASE-26571 URL: https://issues.apache.org/jira/browse/HBASE-26571 Project: HBase Issue Type: Sub-task Components: dependencies, hbase-operator-tools Reporter: Peter Somogyi Assignee: Peter Somogyi -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26572) Upgrade to log4j 2.16.0
Peter Somogyi created HBASE-26572: - Summary: Upgrade to log4j 2.16.0 Key: HBASE-26572 URL: https://issues.apache.org/jira/browse/HBASE-26572 Project: HBase Issue Type: Sub-task Components: dependencies Reporter: Peter Somogyi Assignee: Peter Somogyi Fix For: 3.0.0-alpha-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26573) [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0
Guangxu Cheng created HBASE-26573: - Summary: [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0 Key: HBASE-26573 URL: https://issues.apache.org/jira/browse/HBASE-26573 Project: HBase Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26574) [hbase-operator-tools] Set version as 1.2.0 in master
Guangxu Cheng created HBASE-26574: - Summary: [hbase-operator-tools] Set version as 1.2.0 in master Key: HBASE-26574 URL: https://issues.apache.org/jira/browse/HBASE-26574 Project: HBase Issue Type: Sub-task Reporter: Guangxu Cheng Assignee: Guangxu Cheng -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] First release candidate for hbase 3.0.0-alpha-2 is available for download
We could still add new big features before the last alpha-4 release. After alpha-4 is out, we will cut a branch-3 and then start to stabilize it. For branch-3, we should treat it like branch-2. And after beta1 and beta2, we will cut branch-3.0, to make the final 3.0.0 release. Thanks. Mallikarjun 于2021年12月14日周二 10:14写道: > Not sure if we can add a new feature at this time of the release cycle. > > Since native backup/restore will be part of hbase release for the first > time, these 2 features would be worthwhile to be considered to be part of > the release. > > 1. Add rsgroup support for Backup > Details: https://issues.apache.org/jira/browse/HBASE-26322 > Patch: https://github.com/apache/hbase/pull/3726 > > 2. Add support to take parallel backups > Details: https://issues.apache.org/jira/browse/HBASE-26034 > Patch: https://github.com/apache/hbase/pull/3766 > > > > --- > Mallikarjun > > > On Tue, Dec 14, 2021 at 5:45 AM 张铎(Duo Zhang) > wrote: > > > Thanks Josh! > > > > Will make a new RC1 soon. > > > > Josh Elser 于2021年12月14日周二 04:57写道: > > > > > -1 (binding) > > > > > > Log4j2 CVE mitigation is ineffective due an incorrect `export` in > > > bin/hbase-config.sh. Appears that HBASE-26557 tried to add the > > > mitigation to HBASE_OPTS but added spaces around either side of the > > > equals sign, e.g. `export HBASE_OPTS = ".."`, which is invalid syntax. > > > > > > > > > > > > $ ./bin/start-hbase.sh > > > > > > /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh: > > > > > > line 167: export: `=': not a valid identifier > > > > > > /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh: > > > > > > line 167: export: ` -Dlog4j2.formatMsgNoLookups=true': not a valid > > > identifier > > > > > > /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh: > > > > > > line 167: export: `=': not a valid identifier > > > > > > /Users/jelser/hbase300alpha2rc0/hbase300/hbase-3.0.0-alpha-2/bin/hbase-config.sh: > > > > > > line 167: export: ` -Dlog4j2.formatMsgNoLookups=true': not a valid > > > identifier > > > > > > > > > More naively, and just in plain bash: > > > > > > bash-5.1$ export FOO = "$FOO bar" > > > bash: export: `=': not a valid identifier > > > bash: export: ` bar': not a valid identifier > > > bash-5.1$ echo $FOO > > > > > > > > > I'll post a PR to fix after sending this. > > > > > > The good: > > > * xsums and sigs were OK > > > * Was able to run most unit tests locally > > > * Was able to launch using bin tarball > > > * Everything else looks great so far > > > > > > - Josh > > > > > > On 12/11/21 11:34 AM, Duo Zhang wrote: > > > > Please vote on this Apache hbase release candidate, > > > > hbase-3.0.0-alpha-2RC0 > > > > > > > > The VOTE will remain open for at least 72 hours. > > > > > > > > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 > > > > [ ] -1 Do not release this package because ... > > > > > > > > The tag to be voted on is 3.0.0-alpha-2RC0: > > > > > > > >https://github.com/apache/hbase/tree/3.0.0-alpha-2RC0 > > > > > > > > This tag currently points to git reference > > > > > > > >8bca21b47d7c809a0940aea8ed12dd4d2af12432 > > > > > > > > The release files, including signatures, digests, as well as > CHANGES.md > > > > and RELEASENOTES.md included in this RC can be found at: > > > > > > > >https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC0/ > > > > > > > > Maven artifacts are available in a staging repository at: > > > > > > > > > > > > https://repository.apache.org/content/repositories/orgapachehbase-1472/ > > > > > > > > Artifacts were signed with the 9AD2AE49 key which can be found in: > > > > > > > >https://downloads.apache.org/hbase/KEYS > > > > > > > > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release > > > line. > > > > HBase 3.0.0 includes the following big feature/changes: > > > >Synchronous Replication > > > >OpenTelemetry Tracing > > > >Distributed MOB Compaction > > > >Backup and Restore > > > >Move RSGroup balancer to core > > > >Reimplement sync client on async client > > > >CPEPs on shaded proto > > > >Move the logging framework from log4j to log4j2 > > > > > > > > 3.0.0-alpha-2 contains a critical security fix for addressing the > > log4j2 > > > > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should > upgrade > > > > to 3.0.0-alpha-2 ASAP. > > > > > > > > Notice that this is not a production ready release. It is used to let > > our > > > > users try and test the new major release, to get feedback before the > > > final > > > > GA release is out. > > > > So please do NOT use it in production. Just try it and report back > > > > everything you find unusual. > > > > > > > > And this time we will not include CHANGES.md and RELEASENOTE.md > > > > in our source code, you can find it on the download site. For getting > > > these > > > > two files for old releases, please go to > > > >
[VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
Please vote on this Apache hbase release candidate, hbase-3.0.0-alpha-2RC1 The VOTE will remain open for at least 72 hours. [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 [ ] -1 Do not release this package because ... The tag to be voted on is 3.0.0-alpha-2RC1: https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 This tag currently points to git reference a3ff8e4c812eefab6ad32af45ca449a1395a6510 The release files, including signatures, digests, as well as CHANGES.md and RELEASENOTES.md included in this RC can be found at: https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ Maven artifacts are available in a staging repository at: https://repository.apache.org/content/repositories/orgapachehbase-1473/ Artifacts were signed with the 9AD2AE49 key which can be found in: https://downloads.apache.org/hbase/KEYS 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line. HBase 3.0.0 includes the following big feature/changes: Synchronous Replication OpenTelemetry Tracing Distributed MOB Compaction Backup and Restore Move RSGroup balancer to core Reimplement sync client on async client CPEPs on shaded proto Move the logging framework from log4j to log4j2 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade to 3.0.0-alpha-2 ASAP. Notice that this is not a production ready release. It is used to let our users try and test the new major release, to get feedback before the final GA release is out. So please do NOT use it in production. Just try it and report back everything you find unusual. And this time we will not include CHANGES.md and RELEASENOTE.md in our source code, you can find it on the download site. For getting these two files for old releases, please go to https://archive.apache.org/dist/hbase/ To learn more about Apache hbase, please see http://hbase.apache.org/ Thanks, Your HBase Release Manager
[jira] [Resolved] (HBASE-26554) Introduce a new parameter in jmx servlet to exclude the specific mbean
[ https://issues.apache.org/jira/browse/HBASE-26554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26554. --- Hadoop Flags: Reviewed Resolution: Fixed Pushed to master and branch-2. Thanks [~frostruan] for ocontriburing. Please fill the release note about the change here. > Introduce a new parameter in jmx servlet to exclude the specific mbean > -- > > Key: HBASE-26554 > URL: https://issues.apache.org/jira/browse/HBASE-26554 > Project: HBase > Issue Type: Improvement > Components: metrics >Affects Versions: 2.4.8 >Reporter: ruanhui >Assignee: ruanhui >Priority: Minor > Fix For: 2.6.0, 3.0.0-alpha-3 > > > There are many regionservers serving over a thousand regions in our company, > and the metric load is pretty big. > I tried to exclude some huge mbean like > 'Hadoop:service=HBase,name=RegionServer,sub=Regions' with regex, but didn't > succeed. > So I want to propose a new parameter 'excl' in jmx servlet to exclude the > splecific bean or beans. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26550) Make sure the master is running normally before accepting a balance command
[ https://issues.apache.org/jira/browse/HBASE-26550?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-26550. --- Fix Version/s: 2.5.0 2.6.0 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2.5+. Thanks [~elserj]! > Make sure the master is running normally before accepting a balance command > --- > > Key: HBASE-26550 > URL: https://issues.apache.org/jira/browse/HBASE-26550 > Project: HBase > Issue Type: Bug > Components: Balancer, master >Reporter: Josh Elser >Assignee: Josh Elser >Priority: Minor > Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3 > > > Noticed this in a unit test from [https://github.com/apache/hbase/pull/3851] > I believe this is a result of the new balance() implementation in the Master, > and a client submitting a request to the master before it's completed its > instantiation. Simple fix to avoid the NPE. > {noformat} > Failed after attempts=2, exceptions: > 2021-12-07T18:14:19.203Z, java.io.IOException: java.io.IOException > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:461) > at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:138) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:354) > at > org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:334) > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hbase.master.HMaster.getProcedures(HMaster.java:3335) > at > org.apache.hadoop.hbase.master.ServerManager.areDeadServersInProgress(ServerManager.java:505) > at org.apache.hadoop.hbase.master.HMaster.balance(HMaster.java:1886) > at > org.apache.hadoop.hbase.master.MasterRpcServices.balance(MasterRpcServices.java:740) > at > org.apache.hadoop.hbase.shaded.protobuf.generated.MasterProtos$MasterService$2.callBlockingMethod(MasterProtos.java) > at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:393) > ... 3 more {noformat} -- This message was sent by Atlassian Jira (v8.20.1#820001)
[ANNOUNCE] Apache HBase Thirdparty 4.0.0 released
We've just made a new release of the Apache HBase Thirdparty project. This project is used by the Apache HBase project to encapsulate a number of core dependencies that HBase relies upon ensuring that they are properly isolated from HBase downstream users, e.g. Google Protocol Buffers, Google Guava, and a few others. In 4.0.0 we bump the version of some dependencies. In HBASE-25496 we fixed a problem which may cause NoSuchMethodError, please see HBASE-26492 for more details. And we also shade javax.ws.rs and jackson-jaxrs-json-provider for use with shaded jersey. The release is available as source at https://downloads.apache.org/[1] and as artifacts up in Maven central[2]. Release notes and Changes can be found at [1]. The release was tagged rel/4.0.0. All artifacts are signed with my key 9AD2AE49 which can be found here: https://downloads.apache.org/hbase/KEYS - Your Release Manager [1] https://downloads.apache.org/hbase/hbase-thirdparty-4.0.0/ [2] https://repository.apache.org/service/local/repositories/releases/content/org/apache/hbase/thirdparty/hbase-thirdparty/4.0.0/
Re: [DISCUSS] In flight work to complete before 2.5.0RC0
Hi Andrew and team, The other one I'd like to land for 2.5 is the region visualizer on HBASE-25865. It stalled for most of the year as it required embedding an HTTP/JSON API into the Master's WebUI, and I identified issues with our shading of Jersey and its associated JSR implementations. Since those have been fixed in hbase-thirdparty-4.0.0, I'd like to revive this feature and bring it home to conclusion. There are screenshots of the progress thus far attached to the jira. I've implemented one visualization, a stacked bar chart that illustrates the total storefile size per table per region server. As a stretch-goal (or follow-on), I'd also like to implement a region size histogram. Once we have a couple in place, there should be enough code infrastructure that adding other views will be relatively painless. For now, the browser-side development goes slowly because I have limited JavaScript skills. As mentioned, this feature relies on an HTTP/JSON endpoint being exposed by the WebUI that hosts the visualization. I have implemented the bare minimum required endpoint for the visualization that I have on the sub-task HBASE-25895. While I intend this to be advertised as an internal-only, IA.Private API, it deserves some scrutiny. Thanks, Nick On Thu, Dec 9, 2021 at 4:03 PM Nick Dimiduk wrote: > The OpenTelemetry improvements are tracked as subtasks of HBASE-26419 [0]. > More eyes on the open PRs would be very helpful, and if anyone is > interested in joining the discussion in the CNCF community, you can join > their Slack [1]. One change often depends on the previous, so I have been > working through the list and pushing patches onto a feature branch [2]. > > Thanks, > Nick > > [0]: https://issues.apache.org/jira/browse/HBASE-26419 > [1]: https://cloud-native.slack.com/archives/C0150QF88FL > [2]: > https://github.com/ndimiduk/hbase/tree/26419-otel-semantic-conventions > > On Wed, Dec 8, 2021 at 6:09 PM 张铎(Duo Zhang) > wrote: > >> OpenTracing -> OpenTelemetry :) >> >> For me, I think the OpenTelemetry part is a blocker, we must finish it >> before cutting an RC since the current implementation is already landed on >> branch-2.5 and it breaks some Otel best practises, so we should not >> release >> it out. >> >> Now it is only Nick doing the work and Tak Lon Wu and I reviewing the PRs. >> And I also joined the CNCF slack channel and saw Nick is working hard in >> communication with the Otel community on how to better implement tracing >> in >> HBase, for example, how to trace big scans. >> I would encourage more people in our community to involve so we can make >> progress faster. >> >> Thanks. >> >> Sean Busbey 于2021年12月9日周四 10:02写道: >> >> > If we don't want to wait for HBASE-26543 (fix arg parsing for shell) >> > then we should revert HBASE-24772 from branch-2.5 prior to an RC. >> > >> > >> > On Wed, Dec 8, 2021 at 7:34 PM Andrew Purtell >> wrote: >> > > >> > > As your branch-2.5 RM I am assembling a list of work items that >> should be >> > > completed before a 2.5.0RC0 candidate is submitted for the PMC's >> > > consideration. >> > > >> > > I have so far: >> > > >> > > - OpenTracing span naming convention and coverage improvements. >> > > >> > > - Shell exit code fixes/improvements. >> > > >> > > - The "encryption improvements umbrella". Arguable, but let's include >> it >> > > for now. Can all be resolved as Later if need be. >> > > >> > > Let's discuss what else, if anything, should be on this list, or if >> one >> > or >> > > more of the above items does not constitute a release blocker. I >> consider >> > > incomplete work-in-progress a blocker. Obviously all of the work in >> > > progress should land before release. For WIP, let's also agree on a >> > > definition of done. >> > > >> > > -- >> > > Best regards, >> > > Andrew >> > > >> > > Words like orphans lost among the crosstalk, meaning torn from truth's >> > > decrepit hands >> > >- A23, Crosstalk >> > >> >
[jira] [Created] (HBASE-26575) StoreHotnessProtector may block Replication
Bryan Beaudreault created HBASE-26575: - Summary: StoreHotnessProtector may block Replication Key: HBASE-26575 URL: https://issues.apache.org/jira/browse/HBASE-26575 Project: HBase Issue Type: Bug Reporter: Bryan Beaudreault I'm upgrading from hbase1 to hbase2, and I'm still in my QA environment where load is very low. Even still, I've noticed some bad interaction between Replication and the StoreHotnessProtector. The ReplicationSink collects edits from the WAL and executes them in batches via the normal HTable interface. Despite the name of this property, the max batch sizes are based on "hbase.rpc.rows.warning.threshold" which has a default of 5000. The StoreHotnessProtector defaults to allowing 10 concurrent writes (of 100 columns or more) to a Store, or 20 concurrent "prepares" of said writes. The Prepare part is what causes issues here. When a batch mutate comes in, the RS first takes a lock on all rows in the batch. This happens in HRegion#lockRowsAndBuildMiniBatch, and the writes are recorded as "preparing" in StoreHotnessProtector before acquiring the lock. This recording basically increments a counter, and throws an exception if that counter goes over 20. Back in HRegion#lockRowsAndBuildMiniBatch, the exception is caught and recorded in the results for any items that failed. Any items that succeed continue on to write, unless the write is atomic, in which case it immediately throws an exception. This response gets back to the client, which automatically handles retries. With enough retries, the batch call will eventually succeed because each retry contains fewer and fewer writes to handle. Assuming you have enough retries, this is basically enforcing an automatic chunking of of a batch write into sub-batches of 20. Again, this only affects writes that hit more than 100 columns (by default). At this point I'll say that this in general seems overly aggressive, especially since the StoreHotnessProtector doesn't actually do any checks for actual load on the RS. You could have a totally idle RegionServer and submit a single batch of 100 Puts with 101 columns each – if you don't have at least 5 retries configured, the batch will fail. Back to ReplicationSink, the default batch size is 5000 Puts and the default retries is 4. For a table with wide rows (which might cause replication to try to sink Puts with more than 100 columns), it becomes basically impossible to replicate because the number of retries is not nearly enough to move through a batch of up to 5000, 20 at a time. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] Second release candidate for hbase 3.0.0-alpha-2 is available for download
+1 * Signature: ok * Checksum : ok * Rat check (11.0.11): ok - mvn clean apache-rat:check -D skipTests * Built from source (11.0.11): ok - mvn clean install -D skipTests -DskipTests On Tue, Dec 14, 2021 at 6:16 AM Duo Zhang wrote: > Please vote on this Apache hbase release candidate, > hbase-3.0.0-alpha-2RC1 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase 3.0.0-alpha-2 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 3.0.0-alpha-2RC1: > > https://github.com/apache/hbase/tree/3.0.0-alpha-2RC1 > > This tag currently points to git reference > > a3ff8e4c812eefab6ad32af45ca449a1395a6510 > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/3.0.0-alpha-2RC1/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1473/ > > Artifacts were signed with the 9AD2AE49 key which can be found in: > > https://downloads.apache.org/hbase/KEYS > > 3.0.0-alpha-2 is the second alpha release for our 3.0.0 major release line. > HBase 3.0.0 includes the following big feature/changes: > Synchronous Replication > OpenTelemetry Tracing > Distributed MOB Compaction > Backup and Restore > Move RSGroup balancer to core > Reimplement sync client on async client > CPEPs on shaded proto > Move the logging framework from log4j to log4j2 > > 3.0.0-alpha-2 contains a critical security fix for addressing the log4j2 > CVE-2021-44228. All users who already use 3.0.0-alpha-1 should upgrade > to 3.0.0-alpha-2 ASAP. > > Notice that this is not a production ready release. It is used to let our > users try and test the new major release, to get feedback before the final > GA release is out. > So please do NOT use it in production. Just try it and report back > everything you find unusual. > > And this time we will not include CHANGES.md and RELEASENOTE.md > in our source code, you can find it on the download site. For getting these > two files for old releases, please go to > > https://archive.apache.org/dist/hbase/ > > To learn more about Apache hbase, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager >
[jira] [Resolved] (HBASE-26570) Upgrade to log4j 2.16.0
[ https://issues.apache.org/jira/browse/HBASE-26570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peter Somogyi resolved HBASE-26570. --- Resolution: Done > Upgrade to log4j 2.16.0 > --- > > Key: HBASE-26570 > URL: https://issues.apache.org/jira/browse/HBASE-26570 > Project: HBase > Issue Type: Umbrella > Components: dependencies >Reporter: Peter Somogyi >Assignee: Peter Somogyi >Priority: Major > > Log4j just release version 2.16.0 where jndi is turned off by default. Based > on the release announcement it is not required to fix CVE-2021-44228 but > recommended. > [https://lists.apache.org/thread/d6v4r6nosxysyq9rvnr779336yf0woz4] -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26576) Allow Pluggable Queue to belong to FastPath or normal Balanced Executor
Richard Marscher created HBASE-26576: Summary: Allow Pluggable Queue to belong to FastPath or normal Balanced Executor Key: HBASE-26576 URL: https://issues.apache.org/jira/browse/HBASE-26576 Project: HBase Issue Type: New Feature Components: regionserver, rpc Reporter: Richard Marscher Assignee: Richard Marscher The PluggableQueue implementation added in[ https://github.com/apache/hbase/pull/3522|https://github.com/apache/hbase/pull/3522)] for https://issues.apache.org/jira/browse/HBASE-6908 is only wired up with the BalancedQueueRpcExecutor. Users should be able to decide their queue operates with the FastPathBalancedQueueRpcExecutor optionally via configuration. I plan to submit a PR for this update soon. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26556) IT and Chaos Monkey improvements
[ https://issues.apache.org/jira/browse/HBASE-26556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wellington Chevreuil resolved HBASE-26556. -- Resolution: Fixed Merged into master and cherry-picked into branch-2, branch-2.5, branch-2.4 and branch-2.3. > IT and Chaos Monkey improvements > > > Key: HBASE-26556 > URL: https://issues.apache.org/jira/browse/HBASE-26556 > Project: HBase > Issue Type: Improvement > Components: integration tests >Affects Versions: 3.0.0-alpha-1, 2.5.0, 2.3.7, 2.4.7, 2.4.8, 2.6.0 >Reporter: Wellington Chevreuil >Assignee: Wellington Chevreuil >Priority: Minor > Fix For: 2.5.0, 2.3.8, 2.4.9, 2.6.0 > > > While running the IntegrationTestIngest with slowDeterministic chaos monkey > in our environment, we faced some issues with some of the chaos actions that > required run some linux shell commands, due to the way we set hbase > environment variables and start/stop scripts. > This patch allows for defining extensions of HBaseShellCommandProvider in the > configuration, so that custom commands can be used. It also extends > SlowDeterministicMonkeyFactory to provide the ability to configure the > destructive actions to be executed. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26577) Update ref guide section for IT and Chaos Monkey to explain the additions from HBASE-26556
Wellington Chevreuil created HBASE-26577: Summary: Update ref guide section for IT and Chaos Monkey to explain the additions from HBASE-26556 Key: HBASE-26577 URL: https://issues.apache.org/jira/browse/HBASE-26577 Project: HBase Issue Type: Improvement Reporter: Wellington Chevreuil Assignee: Wellington Chevreuil HBASE-26556 introduced a customisable monkey factory for the slow deterministic policy, as well as made possible for pluggable implementations of the hbase remote shell commands. This is to document how these new features can be used. -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [DISCUSS] Merge HBASE-26067 branch into master, and backport it to base 2 branches
Thanks for your input, Andrew and Nick! Big thank you to Duo for your hands-on-keyboard commitment as well for this whole feature. I am also happy to target 2.x (and not 2.5.x) for the backport. In the interest of getting rid of this feature branch (and the inevitable rebase pains the longer it runs parallel to master), I'd like to move ahead with a concrete plan to merge. 1. Given there was no objection, do folks feel the need for a VOTE? Even if one person would like a VOTE, I'm happy to start that. Please just say so. 2. We have three outstanding PRs for the sake of SFT which are all (IMO) very close to merging (#3851, #3861, and #3942). I think 3851 and 3942 are easy to include and just need one more review cycle. If we feel like we are still far away on 3861, I think we set that aside and revisit it after the feature merge is done. If there are any other concerns, please shout! - Josh On 12/8/21 9:07 PM, Andrew Purtell wrote: +1 for merging to branch-2 (2.6) On Dec 8, 2021, at 6:04 PM, 张铎 wrote: I think here we just want this to be backported to 2.x, not 2.5.x. So thanks Andrew for the quick action. +1 on merging HBASE-26067 to master and backporting to branch-2(2.6.0). Thanks. Andrew Purtell 于2021年12月9日周四 08:45写道: I concur with Nick, but let me help here by branching 2.5 today. It was always going to be somewhat arbitrary a point. On Wed, Dec 8, 2021 at 3:09 PM Nick Dimiduk wrote: Based solely on the comments made to this thread, I would recommend against a merge to branch-2, given that we are very close to 2.5. The points about existing gaps seem like things we're not ready to publish in the impending minor release. Once we have a branch-2.5, this particular concern of mine will be alleviated. Thanks, Nick On Wed, Dec 8, 2021 at 1:37 PM Josh Elser wrote: I was going to wait for some other folks to chime in, but I guess I can be the next one :) Duo, Wellington, and Szabolcs have been doing some excellent work on the storefile tracking (SFT) to a degree that I never expected to see. I remember some of the original "Filesystem re-do" issues on Jira. The idea was exceptional, but the result seemed unreachable. These devs, building on the success of what Zach/Stephen first talked about in HBASE-24749, came up with what I think is an excellent step forward. I've yet to break it via my own testing, but do acknowledge that there's always more work to be done. I think this is at a reasonable place to merge this back into the "mainline" branches from the feature branch (HBASE-26067). I believe this is ready because: 1. The feature is completely opt-in (HBase works the same way by default) 2. There is API to migrate tables into the new SFT implementation 3. There is also API to migrate tables back to the default implementation Some gaps still exist around bulk loading, documentation, snapshots, and recovery tooling, but these are being worked on. In the context of S3, this makes a significantly more compelling offering of HBase by removing the complexity of HBOSS. For HBase in all installations, I think SFT makes more a significantly more "deterministic" way of managing regions/files. +1 from me to merge HBASE-26067 into master and branch-2 - Josh On 12/7/21 10:31 AM, Wellington Chevreuil wrote: Hello everyone, We have been making progress on the alternative way of tracking store files originally proposed by Duo in HBASE-26067. To briefly summarize it for those not following it, this feature introduces an abstraction layer to track store files still used/needed by store engines, allowing for plugging different approaches of identifying store files required by the given store. The design doc describing it in more detail is available here < https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.calrs3kn4d8s . Our main goal within this feature is to avoid the need for using temp files and renames when creating new hfiles (whenever flushing, compacting, splitting/merging or snapshotting). This is made possible by the pluggable tracker implementation labeled "FILE". The current behavior using temp dirs and renames would still be the default approach (labeled "DEFAULT"). This "renameless" approach is appealing for deployments using Amazon S3 Object store file system, where the lack of atomic rename operations imposed the necessity of an additional layer of locking (HBOSS), which combined with the s3a rename operation can have a performance overhead. Some test runs on my employer infrastructure have shown promising results. A pure insertion ycsb run has shown ~6% performance gain on the client writes. Snapshot clone of hundreds of regions table completes in half of the time. There are also improvements in compaction, splits and merges times. Talking with Duo Zhang and Josh Elser in the HBASE-26067 jira, we feel optimistic that the current implementation is in a goo
[jira] [Resolved] (HBASE-26573) [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0
[ https://issues.apache.org/jira/browse/HBASE-26573?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng resolved HBASE-26573. --- Resolution: Resolved > [hbase-operator-tools] Generate CHANGES.md and RELEASENOTES.md for 1.2.0 > > > Key: HBASE-26573 > URL: https://issues.apache.org/jira/browse/HBASE-26573 > Project: HBase > Issue Type: Sub-task >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
Please vote on this Apache hbase operator tools release candidate, hbase-operator-tools-1.2.0RC0 The VOTE will remain open for at least 72 hours. [ ] +1 Release this package as Apache hbase operator tools 1.2.0 [ ] -1 Do not release this package because ... The tag to be voted on is 1.2.0RC0: https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 This tag currently points to git reference 76d68624cebb66ec0e509b0a4c0d96445a601685 The release files, including signatures, digests, as well as CHANGES.md and RELEASENOTES.md included in this RC can be found at: https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2 .0RC0/ Maven artifacts are available in a staging repository at: https://repository.apache.org/content/repositories/orgapachehbase-1474/ Artifacts were signed with the E65E11D40D80DB7C key which can be found in: https://dist.apache.org/repos/dist/release/hbase/KEYS hbase-operator-tools 1.2.0 contains a critical security fix for addressing the log4j2 CVE-2021-44228. All users who already use hbase-operator-tools 1.1.0 should upgrade to hbase-operator-tools 1.2.0 ASAP. To learn more about Apache hbase operator tools, please see http://hbase.apache.org/ Thanks, Your HBase Release Manager
Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
Sorry, some information is missing from the email. Please ignore the email. -- Best Regards, Guangxu Guangxu Cheng 于2021年12月15日周三 11:25写道: > Please vote on this Apache hbase operator tools release candidate, > hbase-operator-tools-1.2.0RC0 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 1.2.0RC0: > > https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 > > This tag currently points to git reference > > 76d68624cebb66ec0e509b0a4c0d96445a601685 > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2 > .0RC0/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1474/ > Artifacts were signed with the E65E11D40D80DB7C key which can be found in: > > https://dist.apache.org/repos/dist/release/hbase/KEYS > > hbase-operator-tools 1.2.0 contains a critical security fix for addressing > the log4j2 > CVE-2021-44228. All users who already use hbase-operator-tools 1.1.0 > should upgrade > to hbase-operator-tools 1.2.0 ASAP. > > To learn more about Apache hbase operator tools, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager > >
[VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
Please vote on this Apache hbase operator tools release candidate, hbase-operator-tools-1.2.0RC0 The VOTE will remain open for at least 72 hours. [ ] +1 Release this package as Apache hbase operator tools 1.2.0 [ ] -1 Do not release this package because ... The tag to be voted on is 1.2.0RC0: https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 This tag currently points to git reference 76d68624cebb66ec0e509b0a4c0d96445a601685 The release files, including signatures, digests, as well as CHANGES.md and RELEASENOTES.md included in this RC can be found at: https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/ Maven artifacts are available in a staging repository at: https://repository.apache.org/content/repositories/orgapachehbase-1474/ Artifacts were signed with the 5EF3A66D57EC647A key which can be found in: https://dist.apache.org/repos/dist/release/hbase/KEYS hbase-operator-tools 1.2.0 contains a critical security fix for addressing the log4j2 CVE-2021-44228. All users who use hbase-operator-tools should upgrade to hbase-operator-tools 1.2.0 ASAP. To learn more about Apache hbase operator tools, please see http://hbase.apache.org/ Thanks, Your HBase Release Manager -- Best Regards, Guangxu
[jira] [Created] (HBASE-26578) CompactedHFilesDischarger should be shutdown when aborting regionserver
Xiaolin Ha created HBASE-26578: -- Summary: CompactedHFilesDischarger should be shutdown when aborting regionserver Key: HBASE-26578 URL: https://issues.apache.org/jira/browse/HBASE-26578 Project: HBase Issue Type: Bug Components: master, regionserver Affects Versions: 2.0.0, 3.0.0-alpha-1 Reporter: Xiaolin Ha Assignee: Xiaolin Ha The CompactedHFilesDischarger chore service should be shutdown when regionserver aborting. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26574) [hbase-operator-tools] Set version as 1.2.0 in master
[ https://issues.apache.org/jira/browse/HBASE-26574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangxu Cheng resolved HBASE-26574. --- Fix Version/s: hbase-operator-tools-1.2.0 Resolution: Resolved > [hbase-operator-tools] Set version as 1.2.0 in master > - > > Key: HBASE-26574 > URL: https://issues.apache.org/jira/browse/HBASE-26574 > Project: HBase > Issue Type: Sub-task >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: hbase-operator-tools-1.2.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
Re: [VOTE] First release candidate for hbase-operator-tools 1.2.0 is available for download
The KEYS file should use download.a.o instead of dist.a.o. For the vote thread it is not a big deal, but please change it when sending the final announcement email, otherwise your email will be rejected by the moderator of the announce mailing list... Guangxu Cheng 于2021年12月15日周三 11:33写道: > Please vote on this Apache hbase operator tools release candidate, > hbase-operator-tools-1.2.0RC0 > > The VOTE will remain open for at least 72 hours. > > [ ] +1 Release this package as Apache hbase operator tools 1.2.0 > [ ] -1 Do not release this package because ... > > The tag to be voted on is 1.2.0RC0: > > https://github.com/apache/hbase-operator-tools/tree/1.2.0RC0 > > This tag currently points to git reference > > 76d68624cebb66ec0e509b0a4c0d96445a601685 > > The release files, including signatures, digests, as well as CHANGES.md > and RELEASENOTES.md included in this RC can be found at: > > > https://dist.apache.org/repos/dist/dev/hbase/hbase-operator-tools-1.2.0RC0/ > > Maven artifacts are available in a staging repository at: > > https://repository.apache.org/content/repositories/orgapachehbase-1474/ > Artifacts were signed with the 5EF3A66D57EC647A key which can be found in: > > https://dist.apache.org/repos/dist/release/hbase/KEYS > > hbase-operator-tools 1.2.0 contains a critical security fix for addressing > the log4j2 > CVE-2021-44228. All users who use hbase-operator-tools should upgrade > to hbase-operator-tools 1.2.0 ASAP. > > To learn more about Apache hbase operator tools, please see > > http://hbase.apache.org/ > > Thanks, > Your HBase Release Manager > > -- > Best Regards, > Guangxu >