Re: [VOTE] The second release condidate for hbase-thirdparty 4.0.0 is available for download

[Duo Zhang]

Ah, Thanks Nick for explaining and thanks Andrew for testing.

We still need one more +1 to close this vote.

Andrew Purtell  于2021年12月7日周二 05:50写道：

> Ok, change my vote to +1 (binding). The hbase-thirdparty build and
> artifacts are good.
>
> > On Dec 6, 2021, at 1:18 PM, Nick Dimiduk  wrote:
> >
> > On Mon, Dec 6, 2021 at 11:49 AM Andrew Purtell 
> wrote:
> >
> >> -1 (binding)
> >>
> >> Checked sums and signature, ok
> >> RAT check passed, ok
> >> Built from source, ok
> >> Built HEAD of master (d9315fa043) with -Dhbase-thirdparty.version=4.0.0,
> >> hbase-http module tests fail
> >>
> >
> > Adoption of this dependency will require changes to master. I had posted
> > necessary changes on https://github.com/apache/hbase/pull/3243 and Duo
> did
> > his own on https://github.com/apache/hbase/pull/3910.
> >
> > [ERROR] Tests run: 17, Failures: 0, Errors: 1, Skipped: 2, Time elapsed:
> >> 2.29 s <<< FAILURE! - in org.apache.hadoop.hbase.http.TestHttpServer
> >> [ERROR] org.apache.hadoop.hbase.http.TestHttpServer.testJersey  Time
> >> elapsed: 0.123 s  <<< ERROR!
> >> java.io.FileNotFoundException: http://localhost:55106/jersey/foo?op=bar
> >> at
> >>
> >>
> sun.net.www.protocol.http.HttpURLConnection.getInputStream0(HttpURLConnection.java:1898)
> >> at
> >>
> >>
> sun.net.www.protocol.http.HttpURLConnection.access$200(HttpURLConnection.java:92)
> >> at
> >>
> >>
> sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1492)
> >> at
> >>
> >>
> sun.net.www.protocol.http.HttpURLConnection$9.run(HttpURLConnection.java:1490)
> >> at java.security.AccessController.doPrivileged(Native Method)
> >> at
> >>
> >>
> java.security.AccessController.doPrivilegedWithCombiner(AccessController.java:784)
> >> at
> >>
> >>
> sun.net.www.protocol.http.HttpURLConnection.getInputStream(HttpURLConnection.java:1489)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.http.HttpServerFunctionalTest.readOutput(HttpServerFunctionalTest.java:248)
> >> at
> >>
> >>
> org.apache.hadoop.hbase.http.TestHttpServer.testJersey(TestHttpServer.java:519)
> >>
> >>
> >>> On Fri, Dec 3, 2021 at 6:23 AM 张铎(Duo Zhang) 
> >>> wrote:
> >>>
> >>> Please vote on this Apache hbase thirdparty release candidate,
> >>> hbase-thirdparty-4.0.0RC1
> >>>
> >>> The VOTE will remain open for at least 72 hours.
> >>>
> >>> [ ] +1 Release this package as Apache hbase thirdparty 4.0.0
> >>> [ ] -1 Do not release this package because ...
> >>>
> >>> The tag to be voted on is 4.0.0RC1:
> >>>
> >>>  https://github.com/apache/hbase-thirdparty/tree/4.0.0RC1
> >>>
> >>> This tag currently points to git reference
> >>>
> >>>  c81cf5c3d10e85f2f5168b09b51ce14468deddbd
> >>>
> >>> The release files, including signatures, digests, as well as CHANGES.md
> >>> and RELEASENOTES.md included in this RC can be found at:
> >>>
> >>>
> >> https://dist.apache.org/repos/dist/dev/hbase/hbase-thirdparty-4.0.0RC1/
> >>>
> >>> Maven artifacts are available in a staging repository at:
> >>>
> >>>
> >> https://repository.apache.org/content/repositories/orgapachehbase-1471/
> >>>
> >>> Artifacts were signed with the 9AD2AE49 key which can be found in:
> >>>
> >>>  https://downloads.apache.org/hbase/KEYS
> >>>
> >>> To learn more about Apache hbase thirdparty, please see
> >>>
> >>>  http://hbase.apache.org/
> >>>
> >>> Thanks,
> >>> Your HBase Release Manager
> >>>
> >>
> >>
> >> --
> >> Best regards,
> >> Andrew
> >>
> >> Words like orphans lost among the crosstalk, meaning torn from truth's
> >> decrepit hands
> >>   - A23, Crosstalk
> >>
>

Re: [DISCUSS] releasing hbase-connector 1.1.0 ?

[Stephen Wu]

Thanks Sean and Duo for your inputs, I have few questions before we
move forward.

1. For Spark 3.0 support, HBASE-25326 seems to already allow building
it with spark-3.0, or doesn't it work?
2. For the upcoming release version, after reviewing the comments on
HBASE-26491, I found two pending releases, 1.1.0 (3 overall item, 0
pending issues) and 1.0.1 (46 issues, 3 pending issues, HBASE-22338,
HBASE-25326, HBASE-16247). should we have a 1.0.1 release?

Thanks,
Stephen


On Tue, Dec 7, 2021 at 12:26 AM 张铎(Duo Zhang)  wrote:
>
> Thank you Tak Lon Wu. This is really a great news.
>
> https://issues.apache.org/jira/browse/HBASE-26491
>
> In this issue, we noticed that the problem has already been fixed but not
> released yet...
>
> It will be good if we could have a 1.1.0 release soon.
>
> Sean Busbey  于2021年12月7日周二 07:45写道：
>
> > That'd be great!
> >
> > I can help with release process.
> >
> > I'd like us to ensure this go around we have convenience binaries compiled
> > against  Spark 3 as well. That's been a long standing need.
> >
> > On Mon, Dec 6, 2021, 16:43 Tak Lon (Stephen) Wu  wrote:
> >
> > > Hi guys,
> > >
> > > While I'm learning how to release a minor version of hbase via
> > > https://hbase.apache.org/book.html#releasing and an example of 2.3.7
> > > via HBASE-26373, I found hbase-connector was last released on 2019
> > > April.
> > >
> > > So, I'm wondering if I could help to release hbase-connector e.g.
> > > 1.1.0 and get familiar with the release process, at the same time, I
> > > may need a mentor/adviser and discuss the steps/tasks about it (my
> > > plan is to create JIRAs like HBASE-26373).
> > >
> > > any comments?
> > >
> > > Thanks,
> > > Stephen
> > >
> >

[jira] [Created] (HBASE-26547) Passing an invalid DURABILITY when creating a table enters an endless loop of retries

[Bryan Beaudreault (Jira)]

Bryan Beaudreault created HBASE-26547:
-

 Summary: Passing an invalid DURABILITY when creating a table 
enters an endless loop of retries
 Key: HBASE-26547
 URL: https://issues.apache.org/jira/browse/HBASE-26547
 Project: HBase
  Issue Type: Bug
Reporter: Bryan Beaudreault


As part of our Hbase2 upgrade, our automation copies the HTableDescriptor from 
a CDH5 cluster into the HBase2 cluster, then kicks off replication. During our 
testing we encountered a misconfigured table, which had a DURABILITY => 
'DEFAULT', when the correct value is 'USE_DEFAULT'.

In hbase 1.x, any invalid value encountered by Durability.valueOf is try/caught 
and results in the default value of USE_DEFAULT. So this misconfiguration 
caused no pain in cdh5.

In hbase 2.x+, the IllegalArgumentException from Durability.valueOf is no 
longer caught. This is probably a good thing, but unfortunately it caused the 
CreateTableProcedure to fail in a way that resulted in an endless loop of 
retries, with no backoff.

This may be a general issue with CreateTableProcedure – there should probably 
be a pre-step which validates the HTableDescriptor and terminally fails if 
invalid.

Additionally, does it make sense to have a backoff on the retry of procedures? 
The vary rapid retry of this procedure actually caused HDFS issues because it 
was creating many thousands of .regioninfo files in rapid succession, enough to 
lag replication and cause DataNodes to be considered bad, which caused 
RegionServers to abort due to failed WAL writes.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26546) hbase-shaded-client missing required thirdparty classes under hadoop 3.3.1

[Bryan Beaudreault (Jira)]

Bryan Beaudreault created HBASE-26546:
-

 Summary: hbase-shaded-client missing required thirdparty classes 
under hadoop 3.3.1
 Key: HBASE-26546
 URL: https://issues.apache.org/jira/browse/HBASE-26546
 Project: HBase
  Issue Type: Bug
Reporter: Bryan Beaudreault


In HBASE-25792, the shaded thirdparty libraries from hadoop were removed from 
the hbase-shaded-client fat jar to satisfy invariant checks. Unfortunately this 
causes users of hbase-shaded-client to fail, because required classes are not 
available at runtime.

The specific failure I'm seeing is when trying to call new Configuration(), 
which results in:

 
 
{code:java}
Caused by: java.lang.NoClassDefFoundError: 
org/apache/hadoop/thirdparty/com/google/common/base/Preconditions   
  at 
org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:430)
   
  at 
org.apache.hadoop.conf.Configuration$DeprecationDelta.(Configuration.java:443)
   
  at org.apache.hadoop.conf.Configuration.(Configuration.java:525){code}
 
 
If you take a look at the hbase-shaded-client fat jar, it contains the 
org.apache.hadoop.conf.Configuration class as you'd expect. If you decompile 
that class (or look at the 3.3.1 source), you'll see that there is an import 
for org.apache.hadoop.thirdparty.com.google.common.base.Preconditions but the 
fat jar does not provide it.
 
One way for clients to get around this is to add an explicit dependency on 
hadoop-shaded-guava, but this is problematic for a few reasons:
 
- it's best practice to use maven-dependency-plugin to disallow declared, 
unused dependencies (which this would be)
- it requires users to continually keep the version of hadoop-shaded-guava 
up-to-date over time.
- it only covers guava, but there is also protobuf and potentially other shaded 
libraries in the future.
 
I think we should remove the exclusion of {{org/apache/hadoop/thirdparty/**/*}} 
from the shading config and instead add that pattern to the allowlist so that 
hbase-shaded-client is all clients need to get started with hbase.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Merge HBASE-26067 branch into master, and backport it to base 2 branches

[Wellington Chevreuil]

Hello everyone,

We have been making progress on the alternative way of tracking store files
originally proposed by Duo in HBASE-26067.

To briefly summarize it for those not following it, this feature introduces
an abstraction layer to track store files still used/needed by store
engines, allowing for plugging different approaches of identifying store
files required by the given store. The design doc describing it in more
detail is available here

.

Our main goal within this feature is to avoid the need for using temp files
and renames when creating new hfiles (whenever flushing, compacting,
splitting/merging or snapshotting). This is made possible by the pluggable
tracker implementation labeled "FILE". The current behavior using temp dirs
and renames would still be the default approach (labeled "DEFAULT").

This "renameless" approach is appealing for deployments using Amazon S3
Object store file system, where the lack of atomic rename operations
imposed the necessity of an additional layer of locking (HBOSS), which
combined with the s3a rename operation can have a performance overhead.

Some test runs on my employer infrastructure have shown promising results.
A pure insertion ycsb run has shown ~6% performance gain on the client
writes. Snapshot clone of hundreds of regions table completes in half of
the time. There are also improvements in compaction, splits and merges
times.

Talking with Duo Zhang and Josh Elser in the HBASE-26067 jira, we feel
optimistic that the current implementation is in a good state to get merged
into master branch, but it would be nice to hear other opinions about it,
before we effectively commit it. Looking forward to hearing some
thoughts/concerns you might have.

Kind regards,
Wellington.

[jira] [Resolved] (HBASE-26538) Should find a way to clear the replication queue for a legacy region_replica_replication peer

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26538.
---
Fix Version/s: HBASE-26233
 Hadoop Flags: Reviewed
   Resolution: Fixed

> Should find a way to clear the replication queue for a legacy 
> region_replica_replication peer
> -
>
> Key: HBASE-26538
> URL: https://issues.apache.org/jira/browse/HBASE-26538
> Project: HBase
>  Issue Type: Sub-task
>  Components: read replicas, Replication
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: HBASE-26233
>
>
> When rolling upgrading, we will delete the legacy region_replica_replication 
> peer. But since the old region servers still use this peer for replicating, 
> we can not delete all the replication queues.
> We need to find a way to deal with these legacy replication queues after 
> upgrading.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Resolved] (HBASE-26539) The default rpc timeout 200ms is too small for replicating meta edits

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-26539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26539.
---
Fix Version/s: HBASE-26233
 Hadoop Flags: Reviewed
   Resolution: Fixed

> The default rpc timeout 200ms is too small for replicating meta edits
> -
>
> Key: HBASE-26539
> URL: https://issues.apache.org/jira/browse/HBASE-26539
> Project: HBase
>  Issue Type: Sub-task
>  Components: read replicas
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: HBASE-26233
>
>
> For most meta edits, we will call refreshStoreFiles, which is time consuming. 
> When running tests in HBASE-26487, it is very easy to timeout and cause the 
> replicating to pause for a while.
> I think for replicating meta edits, we should have a larger timeout value.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] Proc-v2 based snapshot implementation

[Duo Zhang]

Send this email on behalf of Ruanhui(github id frostruan). Not sure why but
he/she can not send email to the mailing list...

Hi all,

As we all know, currently the snapshot in hbase has a few limitations, so I
want to propose a proc-v2 implementation of snapshot.

Here are some related links.

jira
https://issues.apache.org/jira/browse/HBASE-26323

design doc
https://docs.google.com/document/d/1Il_PB1SenXGr1-mmCIWEogxEMeGZe2fpuN3bMbjqiGI/edit

the initial implementation
https://github.com/apache/hbase/pull/3920

If you are interested, please take a look in your free time. Looking
forward to your advice and feedback.

Thanks.

Re: [DISCUSS] releasing hbase-connector 1.1.0 ?

[Duo Zhang]

Thank you Tak Lon Wu. This is really a great news.

https://issues.apache.org/jira/browse/HBASE-26491

In this issue, we noticed that the problem has already been fixed but not
released yet...

It will be good if we could have a 1.1.0 release soon.

Sean Busbey  于2021年12月7日周二 07:45写道：

> That'd be great!
>
> I can help with release process.
>
> I'd like us to ensure this go around we have convenience binaries compiled
> against  Spark 3 as well. That's been a long standing need.
>
> On Mon, Dec 6, 2021, 16:43 Tak Lon (Stephen) Wu  wrote:
>
> > Hi guys,
> >
> > While I'm learning how to release a minor version of hbase via
> > https://hbase.apache.org/book.html#releasing and an example of 2.3.7
> > via HBASE-26373, I found hbase-connector was last released on 2019
> > April.
> >
> > So, I'm wondering if I could help to release hbase-connector e.g.
> > 1.1.0 and get familiar with the release process, at the same time, I
> > may need a mentor/adviser and discuss the steps/tasks about it (my
> > plan is to create JIRAs like HBASE-26373).
> >
> > any comments?
> >
> > Thanks,
> > Stephen
> >
>