[DISCUSS] Reverting hbase rest protobuf package name

[Istvan Toth]

Hi!

While working on HBASE-28725, I realized that in HBase 3+ the REST protobuf
definition files have been moved to hbase-shaded-protobuf, and the package
name has also been renamed.

While I fully agree with the move to using the thirdparty protobuf library
(in fact I'd like to backport that change to 2.x), I think that moving the
.proto files and renaming the package was not a good idea.

The REST interface does not use the HBase patched features of the protobuf
library, and if we want to maintain any pretense that the REST protobuf
encoding is usable by non-java code, then we should not use it in the
future either.

(If we ever decide to use the patched features for performance reasons, we
will need to define new protobuf messages for that anyway)

Protobuf does not use the package name on the wire, so wire compatibility
is not an issue.

In the unlikely case that someone has implemented an independent REST
client that uses protobuf encoding, this will also ensure compatibility
with the 3.0+ .protoc definitions.

My proposal is:

HBASE-28726  Revert REST
protobuf package to org.apache.hadoop.hbase.shaded.rest
*This applies only to branch-3+:*
1. Move the REST .proto files and compiling back to the hbase-rest module
(but use the same protoc compiler that we use now)
2. Revert the package name of the protobuf messages to the original
3. No other changes, we still use the thirdparty protobuf library.

The other issue is that on HBase 2.x the REST client still requires
unshaded protobuf 2.5.0 which brings back all the protobuf library
conflicts that were fixed in 3.0 and by hbase-shaded-client. To fix this,
my proposal is:

HBASE-28725  Use
thirdparty protobuf for REST interface in HBase 2.x
*This applies only to branch-2.x:*
1. Backport the code changes that use the thirdparty protobuf library for
REST to branch-2.x

With these two changes, the REST code would be almost identical on every
branch, easing maintenance.

What do you think ?

Istvan

[jira] [Created] (HBASE-28726) Revert REST protobuf package to org.apache.hadoop.hbase.shaded.rest

[Istvan Toth (Jira)]

Istvan Toth created HBASE-28726:
---

 Summary: Revert REST protobuf package to 
org.apache.hadoop.hbase.shaded.rest
 Key: HBASE-28726
 URL: https://issues.apache.org/jira/browse/HBASE-28726
 Project: HBase
  Issue Type: Bug
  Components: REST
Reporter: Istvan Toth


In Hbase 3+, the package name of the REST messages has been renamed to 
org.apache.hadoop.hbase.shaded.rest from org.apache.hadoop.hbase.rest 

These definitions are only used by REST, and have nothing to do with standard 
HBase RPC communication.

I propose reverting the package name.
We may also want to move the protobuf definitions back to the hbase-rest module.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28725) Use thirdparty protobuf for REST interface in HBase 2.x

[Istvan Toth (Jira)]

Istvan Toth created HBASE-28725:
---

 Summary: Use thirdparty protobuf for REST interface in HBase 2.x
 Key: HBASE-28725
 URL: https://issues.apache.org/jira/browse/HBASE-28725
 Project: HBase
  Issue Type: Improvement
  Components: REST
Reporter: Istvan Toth
Assignee: Istvan Toth


This change has already been done in branch 3+ as part of the protobuf 2.5 
removal,
We just need to backport it to 2.x.

This removes the requirement of having unshaded protobuf 2.5.0 on the 
hbase-rest client classpath.




--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28724) BucketCache.notifyFileCachingCompleted may throw IllegalMonitorStateException

[Wellington Chevreuil (Jira)]

Wellington Chevreuil created HBASE-28724:


 Summary: BucketCache.notifyFileCachingCompleted may throw 
IllegalMonitorStateException 
 Key: HBASE-28724
 URL: https://issues.apache.org/jira/browse/HBASE-28724
 Project: HBase
  Issue Type: Bug
Reporter: Wellington Chevreuil
Assignee: Wellington Chevreuil


If the prefetch thread completes reading the file blocks faster than the bucket 
cache writer threads are able to drain it from the writer queues, we might run 
into a scenario where BucketCache.notifyFileCachingCompleted may throw 
IllegalMonitorStateException, as we can reach [this block of the 
code|https://github.com/wchevreuil/hbase/blob/684964f1c1693d2a0792b7b721c92693d75b4cea/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/bucket/BucketCache.java#L2106].
 I believe the impact is not critical, as the prefetch thread is already 
finishing at that point, but nevertheless, such error in the logs might be 
misleading.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28684) Remove CellWrapper and use ExtendedCell internally in client side data structure

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-28684.
---
Hadoop Flags: Incompatible change,Reviewed  (was: Incompatible change)
  Resolution: Fixed

Pushed to master and branch-3.

Thanks [~Ddupg] for reviewing!

> Remove CellWrapper and use ExtendedCell internally in client side data 
> structure
> 
>
> Key: HBASE-28684
> URL: https://issues.apache.org/jira/browse/HBASE-28684
> Project: HBase
>  Issue Type: Sub-task
>  Components: API, Client
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 3.0.0-beta-2
>
>
> In general, all Cells in HBase are ExtendedCells, we introduce Cell interface 
> is only for preventing user to call some methods which damage the system.
> So I think we should have internal methods which can get ExtendedCell from 
> the client side data structures so we do not need to cast everywhere.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-28707) Backport the code changes in HBASE-28675 to branch-2.x

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28707?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-28707.
---
Fix Version/s: 2.7.0
   2.6.1
   2.5.10
 Hadoop Flags: Reviewed
 Assignee: Duo Zhang
   Resolution: Fixed

Pushed to all branch-2.x.

Thanks [~ndimiduk] for reviewing!

> Backport the code changes in HBASE-28675 to branch-2.x
> --
>
> Key: HBASE-28707
> URL: https://issues.apache.org/jira/browse/HBASE-28707
> Project: HBase
>  Issue Type: Task
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>  Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> For aligning the code between different branches.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The first release candidate for 2.5.9 (RC0) is available

[Duo Zhang]

+1 binding

Checked sigs and sums: Matched
Rat check: Passed
LICENSE and NOTICE: In place
Compatibility report: 100% compatible
Built from source(Temurin-17.0.11+9):
  mvn clean install -DskipTests -Dhadoop.profile=3.0
  Passed
Run UTs(Temurin-17.0.11+9):
  mvn -Dsurefire.firstPartForkCount=1C
-Dsurefire.secondPartForkCount=1 -Dsurefire.rerunFailingTestsCount=3
-Dhadoop.profile=3.0 test -PrunAllTests -fn &>test.log 10}
scan 'cluster_test', {FILTER =>
org.apache.hadoop.hbase.filter.KeyOnlyFilter.new(true)}
flush 'cluster_test'
major_compact 'cluster_test'
disable 'cluster_test'
drop 'cluster_test'

Pankaj Kumar  于2024年7月11日周四 02:29写道：
>
> +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (17.0.11): ok
>  - mvn clean apache-rat:check -D hadoop.profile=3.0
> * Built from source (17.0.11): ok
>  - mvn clean install -D hadoop.profile=3.0 -DskipTests
> * Unit tests pass (17.0.11): ok
>  - mvn clean package -P runSmallTests -D hadoop.profile=3.0
> -Dsurefire.rerunFailingTestsCount=3
>
>
> - Exercised basic hbase shell commands, works fine
> - HBase web UI looks good.
> - Loaded 2M records using YCSB, didn't observe any problem.
>
>
> Thanks & Regards,
> Pankaj
>
>
> On Tue, Jul 9, 2024 at 2:50 AM Andrew Purtell  wrote:
>
> > Please vote on this Apache HBase release candidate, HBase 2.5.9 RC0.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache HBase 2.5.9
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.5.9RC0:
> >
> >   https://github.com/apache/hbase/tree/2.5.9RC0
> >
> > This tag currently points to git reference
> >
> >   2b4959069b9285e0a6a1075a88d9f0d4a315ea56
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.5.9RC0/
> >
> > The API compatibility report comparing this RC to the previous release
> > can be found at:
> >
> >
> >
> > https://dist.apache.org/repos/dist/dev/hbase/2.5.9RC0/api_compare_2.5.8_to_2.5.9RC0.html
> >
> > There are no binary or source compatibility problems.
> >
> > Maven artifacts are available in a staging repository at:
> >
> >   https://repository.apache.org/content/repositories/orgapachehbase-1548/
> >
> > Maven artifacts for hadoop3 are available in a staging repository at:
> >
> >   https://repository.apache.org/content/repositories/orgapachehbase-1549/
> >
> > Artifacts were signed with the 0xD5365CCD key which can be found in:
> >
> >   https://downloads.apache.org/hbase/KEYS
> >
> > To learn more about Apache hbase, please see
> >
> >   http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
> >

[jira] [Resolved] (HBASE-28665) WALs not marked closed when there are errors in closing WALs

[Viraj Jasani (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28665?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Viraj Jasani resolved HBASE-28665.
--
Fix Version/s: 2.7.0
   2.6.1
   2.5.10
 Hadoop Flags: Reviewed
   Resolution: Fixed

> WALs not marked closed when there are errors in closing WALs
> 
>
> Key: HBASE-28665
> URL: https://issues.apache.org/jira/browse/HBASE-28665
> Project: HBase
>  Issue Type: Bug
>  Components: wal
>Affects Versions: 2.5.8
>Reporter: Kiran Kumar Maturi
>Assignee: Kiran Kumar Maturi
>Priority: Minor
>  Labels: pull-request-available
> Fix For: 2.7.0, 2.6.1, 2.5.10
>
>
> In our production clusters we have observed that when WAL close fails It 
> causes the the oldWAL files not marked as close and not letting them cleaned. 
> When a WAL close fails in closeWriter it increments the error count. 
> {code:java}
> Span span = Span.current();
>  try {
>   span.addEvent("closing writer");
>   writer.close();
>   span.addEvent("writer closed");
> } catch (IOException ioe) {
>   int errors = closeErrorCount.incrementAndGet();
>   boolean hasUnflushedEntries = isUnflushedEntries();
>   if (syncCloseCall && (hasUnflushedEntries || (errors > 
> this.closeErrorsTolerated))) {
> LOG.error("Close of WAL " + path + " failed. Cause=\"" + 
> ioe.getMessage() + "\", errors="
>   + errors + ", hasUnflushedEntries=" + hasUnflushedEntries);
> throw ioe;
>   }
>   LOG.warn("Riding over failed WAL close of " + path
> + "; THIS FILE WAS NOT CLOSED BUT ALL EDITS SYNCED SO SHOULD BE OK", 
> ioe);
> }
> {code}
> When there are errors in closing WAL only twice doReplaceWALWriter enters 
> this code block
> {code:java}
> if (isUnflushedEntries() || closeErrorCount.get() >= 
> this.closeErrorsTolerated) {
>   try {
> closeWriter(this.writer, oldPath, true);
>   } finally {
> inflightWALClosures.remove(oldPath.getName());
>   }
> }
> {code}
>  as we don't mark them closed here like we do it here 
>   
> {code:java}
>   Writer localWriter = this.writer;
>   closeExecutor.execute(() -> {
> try {
>   closeWriter(localWriter, oldPath, false);
> } catch (IOException e) {
>   LOG.warn("close old writer failed", e);
> } finally {
>   // call this even if the above close fails, as there is no 
> other chance we can set
>   // closed to true, it will not cause big problems.
>  {color:red} markClosedAndClean(oldPath);{color}
>   inflightWALClosures.remove(oldPath.getName());
> }
>   });
> {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)