[jira] [Resolved] (HBASE-27440) metrics method removeHistogramMetrics trigger serious memory leak

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27440?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-27440.
---
Fix Version/s: 2.6.0
   3.0.0-alpha-4
   2.5.2
   2.4.16
 Hadoop Flags: Reviewed
   Resolution: Fixed

Pushed to branch-2.4+.

Thanks [~tianliuliu] for contributing!

> metrics method removeHistogramMetrics trigger serious memory leak
> -
>
> Key: HBASE-27440
> URL: https://issues.apache.org/jira/browse/HBASE-27440
> Project: HBase
>  Issue Type: Bug
>  Components: metrics, regionserver
>Affects Versions: 2.4.15
>Reporter: Liuhe Tian
>Assignee: Liuhe Tian
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.2, 2.4.16
>
>
> the method removeHistogramMetrics will cause the metricsMap leak!
> because the put op in the map is the baseName but the remove is not , the 
> origin code add the histogramSuffixe follow the baseName when remove.
> !https://user-images.githubusercontent.com/13758677/196915944-71197086-2d53-4a85-8b5e-0a61441e8f64.png!
> !https://user-images.githubusercontent.com/13758677/196916259-196e3e8a-cde0-4989-8b64-de13b860522d.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Resolved] (HBASE-27438) Make release scripts can generate CHANGES.md and RELEASENOTES.md for 3.x

[Duo Zhang (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-27438?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-27438.
---
Fix Version/s: 3.0.0-alpha-4
 Hadoop Flags: Reviewed
   Resolution: Fixed

Merged to master.

Thanks [~zghao] for reviewing!

> Make release scripts can generate CHANGES.md and RELEASENOTES.md for 3.x
> 
>
> Key: HBASE-27438
> URL: https://issues.apache.org/jira/browse/HBASE-27438
> Project: HBase
>  Issue Type: Improvement
>  Components: community, scripts
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-4
>
>
> For 3.x, we do not have CHANGES.md and RELEASENOTES.md in source control, so 
> we need to find a new way to generate it and commit it to dist.a.o.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [VOTE] The second release candidate for HBase 2.5.1 (RC1) is available

[Peter Somogyi]

+1 (binding)

* Signature: ok
* Checksum : ok
* Rat check (1.8.0_302): ok
  - mvn clean apache-rat:check
* Built from source (1.8.0_302): ok
  - mvn clean install -DskipTests
* Unit tests pass (1.8.0_302): ok
  - mvn package -P runAllTests -Dsurefire.rerunFailingTestsCount=3
* Changes, release notes, compatibility report: ok


On Fri, Oct 21, 2022 at 3:26 PM Nick Dimiduk  wrote:

> +1 (binding)
>
> * Signature: ok
> * Checksum : ok
> * Rat check (1.8.0_292): ok
>  - mvn clean apache-rat:check -D skipTests
> * Built from source (1.8.0_292): ok
>  - mvn clean install -D skipTests -DskipTests
> * Built from source (11.0.11): ok
>  - mvn clean install -D skipTests -D hadoop.profile=3.0 -DskipTests
> * Reviewed API Compatibility summary: ok
> * Reviewed nightly Jenkins: ok
> * Compiled tag vs Hadoop 3.3.4, launched pseudo-distributed
> cluster, ltt -write 7:100:5 -num_keys 10, ltt -read 100:10
> -num_keys 10: ok
>
>
> On Tue, Oct 18, 2022 at 2:47 AM Andrew Purtell 
> wrote:
> >
> > Please vote on this Apache HBase release candidate, hbase-2.5.1RC0.
> >
> > The VOTE will remain open for at least 72 hours.
> >
> > [ ] +1 Release this package as Apache HBase 2.5.1
> > [ ] -1 Do not release this package because ...
> >
> > The tag to be voted on is 2.5.1RC1:
> >
> >   https://github.com/apache/hbase/tree/2.5.1RC1
> >
> > This tag currently points to git reference 894ac15342.
> >
> > The release files, including signatures, digests, as well as CHANGES.md
> > and RELEASENOTES.md included in this RC can be found at:
> >
> >   https://dist.apache.org/repos/dist/dev/hbase/2.5.1RC1/
> >
> > The compatibility report can be found here:
> >
> >
> >
> https://dist.apache.org/repos/dist/dev/hbase/2.5.1RC1/api_compare_2.5.0_to_2.5.1RC1.html
> >
> > All changes are in line with our compatibility guidelines, except one
> > change where we intentionally removed a constant field in a public
> > interface. The change was required to resolve deadlocks during static
> > initialization under certain conditions. Refer to HBASE-27381 for more
> > details.
> >
> > Maven artifacts are available in a staging repository at:
> >
> >
> https://repository.apache.org/content/repositories/orgapachehbase-1499/
> >
> > Artifacts were signed with the 0xD5365CCD key which can be found in:
> >
> >   https://downloads.apache.org/hbase/KEYS
> >
> > To learn more about Apache HBase, please see
> >
> >   http://hbase.apache.org/
> >
> > Thanks,
> > Your HBase Release Manager
>

Re: HBase 2.4.x + Spark 3.3

[Lars Francke]

Hi Andrew,

okay, we'll try that and will report back when/if we get it working.

Cheers,
Lars

On Thu, Oct 20, 2022 at 2:29 AM Andrew Purtell  wrote:

> No, that is insufficient. HBase must be recompiled against Hadoop 3 first
>
> cd /path/to/hbase
> mvn clean install assembly:single -DskipTests -Dhadoop.profile=3.0
> -Dhadoop-three.version=XXX
>
> Then once the results are in your local maven cache or nexus instance, you
> can compile Spark as indicated.
>
>
> On Tue, Oct 18, 2022 at 11:39 PM Lars Francke 
> wrote:
>
> > Hi Andrew,
> >
> > thanks for the reply.
> > I should have been more specific: We only tried to compile the "client"
> > part that's used in Spark itself and we used the proper versions
> >
> > mvn -Dspark.version=XXX -Dscala.version=XXX -Dhadoop-three.version=XXX
> > -Dscala.binary.version=XXX -Dhbase.version=XXX clean package
> >
> > I assume that should pull in the correct dependencies but I have to admit
> > that I didn't check, took it straight from the readme.
> > We wanted to try the server bit for the RegionServers afterwards but
> didn't
> > even get to it yet.
> >
> > We have this on our radar though and might try to work through those
> issues
> > at some point.
> > If we get started on that I'll ping the list.
> >
> > Cheers,
> > Lars
> >
> > On Wed, Oct 19, 2022 at 1:41 AM Andrew Purtell 
> > wrote:
> >
> > > Out of the box use is going to be problematic without recompiling HBase
> > for
> > > Hadoop 3. Spark 3.3 ships with Hadoop 3.3.2. Apache HBase 2.4.x (and
> all
> > > 2.x) releases are compiled against Hadoop 2. Link errors
> (ClassNotFound,
> > > NoClassDef, etc) I think are to be expected because the class
> hierarchies
> > > of various Hadoop things have been incompatibly changed in 3.x releases
> > > relative to 2.x. This is not unreasonable. Semantic versioning suggests
> > > breaking changes can be expected in a major version increment.
> > >
> > > Users probably need to do a holistic (or hermetic, if you prefer) build
> > of
> > > their bill of materials before testing it or certainly before shipping
> > it.
> > > Build your HBase for the version of Hadoop you are actually shipping it
> > > with, as opposed to whatever the upstream project picks as a default
> > build
> > > target. They are called "convenience binaries" by the project and the
> > > Foundation for a reason. Convenience may vary according to your
> > > circumstances. When HBase finally ships builds compiled against Hadoop
> 3
> > by
> > > default, anyone still using 2.x in production will face the same
> problem
> > > (in reverse). The Phoenix project also faces this issue for what it's
> > > worth. Their readme and build instructions walk users through
> rebuilding
> > > HBase using -Dhadoop.profile=3.0 as a first step as well.
> > >
> > >
> > > On Mon, Oct 17, 2022 at 1:52 PM Lars Francke 
> > > wrote:
> > >
> > > > Hi everyone,
> > > >
> > > > we've just recently tried getting the HBase Spark connector running
> > > against
> > > > Spark 3.3 and HBase 2.4.x and failed miserably. It was a mess of
> Scala
> > > and
> > > > Java issues, classpath, NoClassDef etc.
> > > >
> > > > The trauma is too recent for me to dig up the details but if needed I
> > can
> > > > ;-)
> > > >
> > > > For now I'm just wondering if anyone has succeeded using this
> > > combination?
> > > >
> > > > Cheers,
> > > > Lars
> > > >
> > >
> > >
> > > --
> > > Best regards,
> > > Andrew
> > >
> > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > It's what we’ve earned
> > > Welcome, apocalypse, what’s taken you so long?
> > > Bring us the fitting end that we’ve been counting on
> > >- A23, Welcome, Apocalypse
> > >
> >
>
>
> --
> Best regards,
> Andrew
>
> Unrest, ignorance distilled, nihilistic imbeciles -
> It's what we’ve earned
> Welcome, apocalypse, what’s taken you so long?
> Bring us the fitting end that we’ve been counting on
>- A23, Welcome, Apocalypse
>

Re: HBase 2.4.x + Spark 3.3

[Wei-Chiu Chuang]

This looks similar to what we are seeing with Ozone running w/ Spark 3.3
HDDS-6926 

My guess is Spark 3.3 switched to the shaded Hadoop client jar, where
protobuf message classes are shaded, and somehow that breaks applications
using old Hadoop classes.

If you can share some stack traces that would be great.


On Mon, Oct 24, 2022 at 11:26 AM Lars Francke 
wrote:

> Hi Andrew,
>
> okay, we'll try that and will report back when/if we get it working.
>
> Cheers,
> Lars
>
> On Thu, Oct 20, 2022 at 2:29 AM Andrew Purtell 
> wrote:
>
> > No, that is insufficient. HBase must be recompiled against Hadoop 3 first
> >
> > cd /path/to/hbase
> > mvn clean install assembly:single -DskipTests -Dhadoop.profile=3.0
> > -Dhadoop-three.version=XXX
> >
> > Then once the results are in your local maven cache or nexus instance,
> you
> > can compile Spark as indicated.
> >
> >
> > On Tue, Oct 18, 2022 at 11:39 PM Lars Francke 
> > wrote:
> >
> > > Hi Andrew,
> > >
> > > thanks for the reply.
> > > I should have been more specific: We only tried to compile the "client"
> > > part that's used in Spark itself and we used the proper versions
> > >
> > > mvn -Dspark.version=XXX -Dscala.version=XXX -Dhadoop-three.version=XXX
> > > -Dscala.binary.version=XXX -Dhbase.version=XXX clean package
> > >
> > > I assume that should pull in the correct dependencies but I have to
> admit
> > > that I didn't check, took it straight from the readme.
> > > We wanted to try the server bit for the RegionServers afterwards but
> > didn't
> > > even get to it yet.
> > >
> > > We have this on our radar though and might try to work through those
> > issues
> > > at some point.
> > > If we get started on that I'll ping the list.
> > >
> > > Cheers,
> > > Lars
> > >
> > > On Wed, Oct 19, 2022 at 1:41 AM Andrew Purtell 
> > > wrote:
> > >
> > > > Out of the box use is going to be problematic without recompiling
> HBase
> > > for
> > > > Hadoop 3. Spark 3.3 ships with Hadoop 3.3.2. Apache HBase 2.4.x (and
> > all
> > > > 2.x) releases are compiled against Hadoop 2. Link errors
> > (ClassNotFound,
> > > > NoClassDef, etc) I think are to be expected because the class
> > hierarchies
> > > > of various Hadoop things have been incompatibly changed in 3.x
> releases
> > > > relative to 2.x. This is not unreasonable. Semantic versioning
> suggests
> > > > breaking changes can be expected in a major version increment.
> > > >
> > > > Users probably need to do a holistic (or hermetic, if you prefer)
> build
> > > of
> > > > their bill of materials before testing it or certainly before
> shipping
> > > it.
> > > > Build your HBase for the version of Hadoop you are actually shipping
> it
> > > > with, as opposed to whatever the upstream project picks as a default
> > > build
> > > > target. They are called "convenience binaries" by the project and the
> > > > Foundation for a reason. Convenience may vary according to your
> > > > circumstances. When HBase finally ships builds compiled against
> Hadoop
> > 3
> > > by
> > > > default, anyone still using 2.x in production will face the same
> > problem
> > > > (in reverse). The Phoenix project also faces this issue for what it's
> > > > worth. Their readme and build instructions walk users through
> > rebuilding
> > > > HBase using -Dhadoop.profile=3.0 as a first step as well.
> > > >
> > > >
> > > > On Mon, Oct 17, 2022 at 1:52 PM Lars Francke  >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > we've just recently tried getting the HBase Spark connector running
> > > > against
> > > > > Spark 3.3 and HBase 2.4.x and failed miserably. It was a mess of
> > Scala
> > > > and
> > > > > Java issues, classpath, NoClassDef etc.
> > > > >
> > > > > The trauma is too recent for me to dig up the details but if
> needed I
> > > can
> > > > > ;-)
> > > > >
> > > > > For now I'm just wondering if anyone has succeeded using this
> > > > combination?
> > > > >
> > > > > Cheers,
> > > > > Lars
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > It's what we’ve earned
> > > > Welcome, apocalypse, what’s taken you so long?
> > > > Bring us the fitting end that we’ve been counting on
> > > >- A23, Welcome, Apocalypse
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> >- A23, Welcome, Apocalypse
> >
>