Re: [DISCUSS] Move our official slack channel to the one in the-asf.slack.com

[Wei-Chiu Chuang]

+1

On Sun, Jul 7, 2024 at 8:07 AM Duo Zhang  wrote:

> As I mentioned in another thread, now slack will hide the comments
> before 90 days in the current apache-hbase.slack.com, which is really
> not good for finding useful discussions.
>
> According to the documentation here
>
> https://infra.apache.org/slack.html
>
> We could invite people which do not have at apache dot org email
> address as a guest to the slack channel, so there is no concerns about
> only committers can join the channel.
>
> Thoughts?
>
> Thanks.
>

Re: Build failure on Apple (M1/M3) for hbase-common package for io.opentelemetry:opentelemetry-context, while it works on Intel Mac

[Wei-Chiu Chuang]

Compare the local maven repo settings:
~/.m2/settings.xml

For me, I mostly use my employer's maven proxy, which occasionally causes
build issues if I am not behind the corporate VPN.
Maybe there's something different in your maven repo settings between the
two Macs.

You could also manually download the dependency and then install the jars.

On Wed, May 1, 2024 at 1:11 PM Mihir Monani  wrote:

> Thanks for checking guys.I tried different maven/java versions but none of
> them solved my issue. I also tried running maven in debug mode but it's not
> throwing any error or warning/exceptions.
>
> Do you have any recommendations on how I debug/solve this?
>
> On Tue, Apr 30, 2024 at 6:06 PM Bryan Beaudreault  >
> wrote:
>
> > I also routinely build all branches of hbase on an apple M3, using
> hadoop3,
> > and Java 11+. I believe I’ve also built with java8 at some point, but
> don’t
> > quote me on that because we largely don’t use java8 at my company.
> >
> >
> > On Tue, Apr 30, 2024 at 4:48 PM Wei-Chiu Chuang 
> > wrote:
> >
> > > I am on Apple M3, checked out HBase master branch,
> > > my JDK is Zulu 1.8.0_392 and Maven 3.9.6.
> > >
> > > I am able to build without modifying HBase code.
> > >
> > > On Tue, Apr 30, 2024 at 1:39 PM Mihir Monani 
> > > wrote:
> > >
> > > > Also I tried the build with OpenJDK 8 aarchx64 JDK version and x64
> JDK
> > > > version, and it fails for both of them.
> > > >
> > > > If I explicitly add dependency for hbase-common/pom.xml, then it
> passes
> > > for
> > > > hbase-common.
> > > >
> > > > *diff --git a/hbase-common/pom.xml b/hbase-common/pom.xml*
> > > >
> > > > *index 0723298926..348c3a4bd4 100644*
> > > >
> > > > *--- a/hbase-common/pom.xml*
> > > >
> > > > *+++ b/hbase-common/pom.xml*
> > > >
> > > > @@ -93,6 +93,10 @@
> > > >
> > > >io.opentelemetry
> > > >
> > > >opentelemetry-api
> > > >
> > > >  
> > > >
> > > > +
> > > >
> > > > +  io.opentelemetry
> > > >
> > > > +  opentelemetry-context
> > > >
> > > > +
> > > >
> > > >  
> > > >
> > > >io.opentelemetry
> > > >
> > > >opentelemetry-semconv
> > > >
> > > > It still fails for hbase-server package with below error
> > > >
> > > > [*INFO*] Apache HBase - Server ..
> > > > *FAILURE* [  0.455
> > > > s]
> > > >
> > > > [*INFO*]
> > > >
> > >
> >
> **
> > > >
> > > > [*INFO*] *BUILD FAILURE*
> > > >
> > > > [*INFO*]
> > > >
> > >
> >
> **
> > > >
> > > > [*INFO*] Total time:  01:32 min
> > > >
> > > > [*INFO*] Finished at: 2024-04-30T13:31:54-07:00
> > > >
> > > > [*INFO*]
> > > >
> > >
> >
> **
> > > >
> > > > [*ERROR*] Failed to execute goal
> > > > org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run *(generate)*
> on
> > > > project hbase-server: *An Ant BuildException has occured:
> > > > java.lang.NoClassDefFoundError: javax/el/ELException*
> > > >
> > > > [*ERROR*] *around Ant part ... > > uriroot="${src.webapps}/master"
> > > > outputdir="${generated.sources}/java"
> > > > package="org.apache.hadoop.hbase.generated.master"
> > > > webxml="${build.webapps}/master/WEB-INF/web.xml" />... @ 17:190 in
> > > >
> > > >
> > >
> >
> /Users/mmonani/git/sfdc/bigdata-packaging/hbase/hbase-server/target/antrun/build-main.xml*:
> > > > javax.el.ELException
> > > >
> > > > [*ERROR*] -> *[Help 1]*
> > > >
> > > >
> > > >
> > > > Does anyone have any issues with Apple Silicon MacBook M1/M3 ?
> > > >
> > > > On Tue, Apr 30, 2024 at 11:30 AM Mihir Monani <
> monani.mi...@gmail.com>
> > > > wrote:
> > >

Re: Build failure on Apple (M1/M3) for hbase-common package for io.opentelemetry:opentelemetry-context, while it works on Intel Mac

[Wei-Chiu Chuang]

I am on Apple M3, checked out HBase master branch,
my JDK is Zulu 1.8.0_392 and Maven 3.9.6.

I am able to build without modifying HBase code.

On Tue, Apr 30, 2024 at 1:39 PM Mihir Monani  wrote:

> Also I tried the build with OpenJDK 8 aarchx64 JDK version and x64 JDK
> version, and it fails for both of them.
>
> If I explicitly add dependency for hbase-common/pom.xml, then it passes for
> hbase-common.
>
> *diff --git a/hbase-common/pom.xml b/hbase-common/pom.xml*
>
> *index 0723298926..348c3a4bd4 100644*
>
> *--- a/hbase-common/pom.xml*
>
> *+++ b/hbase-common/pom.xml*
>
> @@ -93,6 +93,10 @@
>
>io.opentelemetry
>
>opentelemetry-api
>
>  
>
> +
>
> +  io.opentelemetry
>
> +  opentelemetry-context
>
> +
>
>  
>
>io.opentelemetry
>
>opentelemetry-semconv
>
> It still fails for hbase-server package with below error
>
> [*INFO*] Apache HBase - Server ..
> *FAILURE* [  0.455
> s]
>
> [*INFO*]
> **
>
> [*INFO*] *BUILD FAILURE*
>
> [*INFO*]
> **
>
> [*INFO*] Total time:  01:32 min
>
> [*INFO*] Finished at: 2024-04-30T13:31:54-07:00
>
> [*INFO*]
> **
>
> [*ERROR*] Failed to execute goal
> org.apache.maven.plugins:maven-antrun-plugin:3.1.0:run *(generate)* on
> project hbase-server: *An Ant BuildException has occured:
> java.lang.NoClassDefFoundError: javax/el/ELException*
>
> [*ERROR*] *around Ant part ... outputdir="${generated.sources}/java"
> package="org.apache.hadoop.hbase.generated.master"
> webxml="${build.webapps}/master/WEB-INF/web.xml" />... @ 17:190 in
>
> /Users/mmonani/git/sfdc/bigdata-packaging/hbase/hbase-server/target/antrun/build-main.xml*:
> javax.el.ELException
>
> [*ERROR*] -> *[Help 1]*
>
>
>
> Does anyone have any issues with Apple Silicon MacBook M1/M3 ?
>
> On Tue, Apr 30, 2024 at 11:30 AM Mihir Monani 
> wrote:
>
> > If I add *io.opentelemetry:opentelemetry-context* as dependency in
> > hbase-common/pom.xml, it works fine.
> >
> > I am more confused on why this works with Intel MacBook but doesn't work
> > with M1/M3 MacBook.
> >
> > On Tue, Apr 30, 2024 at 11:28 AM Mihir Monani 
> > wrote:
> >
> >> Hi
> >>
> >> I am trying to build hbase (mvn clean install -DskipTests) and it fails
> >> with the below error in hbase-common package.  (Maven version : 3.8.8,
> Java
> >> Version : JDK openjdk version "1.8.0_401")
> >>
> >> [*ERROR*] Failed to execute goal
> >> org.apache.maven.plugins:maven-compiler-plugin:3.8.1:compile
> >> *(default-compile)* on project hbase-common: *Compilation failure*:
> >> Compilation failure:
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:[25,32]
> >> package io.opentelemetry.context does not exist
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:[26,32]
> >> package io.opentelemetry.context does not exist
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/trace/TraceUtil.java:[67,52]
> >> cannot find symbol
> >>
> >> [*ERROR*]   symbol:   class Context
> >>
> >> [*ERROR*]   location: class org.apache.hadoop.hbase.trace.TraceUtil
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/trace/HFileContextAttributesBuilderConsumer.java:[29,32]
> >> package io.opentelemetry.context does not exist
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/hfile/trace/HFileContextAttributesBuilderConsumer.java:[56,23]
> >> cannot find symbol
> >>
> >> [*ERROR*]   symbol:   class ContextKey
> >>
> >> [*ERROR*]   location: class
> >> org.apache.hadoop.hbase.io
> .hfile.trace.HFileContextAttributesBuilderConsumer
> >>
> >> [*ERROR*]
> >>
> /Users/mmonani/git/apache/hbase/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java:[421,55]
> >> cannot find symbol
> >>
> >> When I checked dependency tree difference for hbase-common package,
> Apple
> >> Silicon Mac doesn't download io.opentelemetry:opentelemetry-context
> while
> >> Intel Mac downloads it as sub dependency
> >> of io.opentelemetry:opentelemetry-api
> >>
> >> Any suggestions on how to fix this?
> >>
> >> Mihir
> >>
> >
> >
> > --
> > Mihir Monani
> > (+1)-415-309-6297
> >
>
>
> --
> Mihir Monani
> (+1)-415-309-6297
>

Re: Considering deprecation and removal of XZ compression (hbase-compression-xz)

[Wei-Chiu Chuang]

+1 same here. gzip/lzo in the past, Snappy or zstd now.

On Tue, Apr 2, 2024 at 7:50 PM 张铎(Duo Zhang)  wrote:

> For me I've never seen people actually use the xz compression.
>
> For size, usually people will choose gzip, and for speed, in the past
> people will choose lzo and now they choose snappy or zstd.
>
> So for me I prefer we just deprecated the xz compression immediately
> and remove it 2.6.0.
>
> Thanks.
>
> Andrew Purtell  于2024年4月2日周二 08:02写道：
> >
> > Red Hat filed CVE-2024-3094 late last week on 2024-03-29. This implicates
> > recent releases of the native liblzma library as a vector for malicious
> > code.
> >
> > This is not the pure Java version that we depend upon for HBase's support
> > for the LZMA algorithm (
> >
> https://github.com/apache/hbase/tree/master/hbase-compression/hbase-compression-xz
> ).
> > We depend on version 1.9 of xz-java, which was published in 2021, well
> > before maintenance changes in the project and the involvement of a person
> > who is now believed to be a malicious actor. Projects like HBase that
> > depend on xz-java have no reason to be concerned about the issues
> affecting
> > the native xz library.
> >
> > How the backdoor was introduced calls into question the trustworthiness
> and
> > viability of the XZ project. GitHub has disabled all repositories related
> > to XZ and liblzma, even xz-java. The webpage for XZ and xz-java is down.
> > The open source software community is responding vigorously.
> CVE-2024-3094
> > has a CVSS score 10, the highest possible score. Your security team may
> > become interested in HBase because of hbase-compression-xz's dependency
> on
> > xz-java. It is likely any discovered dependency on any LZMA
> implementation
> > will at least raise questions.
> >
> > For now xz-java remains available in Maven central. (See
> > https://central.sonatype.com/artifact/org.tukaani/xz/versions) We may
> have
> > no choice but to immediately remove hbase-compression-xz if Maven blocks
> or
> > drops xz-java too, because that will break our builds.
> >
> > There is no immediate cause for concern. Still, we believe XZ compression
> > provides little to no value over more modern alternatives, like
> ZStandard,
> > that can also achieve similar compression ratios. XZ, and alternatives
> like
> > ZStandard with the compression level set to a high value, are also
> suitable
> > only for archival use cases and unsuitable for compression of flush files
> > or for use in minor compactions. Given how niche any use of XZ
> > compression could
> > be, we are wondering if there are actually any users of it.
> >
> > If we have no users of hbase-compression-xz, then it provides little to
> no
> > value and continued maintenance of hbase-compression-xz given the issues
> > with its dependency does not make sense.
> >
> > Do you use XZ compression, or are you planning to?
> >
> > If we deprecate XZ compression immediately and then remove it in 2.6,
> would
> > this present a problem? In a private discussion we reached consensus on
> > this approach, but, of course, that is not yet a plan, and something that
> > could easily change based on feedback.
> >
> > From https://nvd.nist.gov/vuln/detail/CVE-2024-3094:
> > "Malicious code was discovered in the upstream tarballs of xz, starting
> > with version 5.6.0. Through a series of complex obfuscations, the liblzma
> > build process extracts a prebuilt object file from a disguised test file
> > existing in the source code, which is then used to modify specific
> > functions in the liblzma code. This results in a modified liblzma library
> > that can be used by any software linked against this library,
> intercepting
> > and modifying the data interaction with this library."
> >
> > --
> > Best regards,
> > Andrew
>

Re: [ANNOUNCE] New HBase committer Istvan Toth

[Wei-Chiu Chuang]

congrats!

On Tue, Apr 2, 2024 at 7:04 AM Pankaj Kumar 
wrote:

> Congratulations Istvan, welcome !!
>
> Regards,
> Pankaj
>
>
> On Tue, Apr 2, 2024 at 4:53 PM Duo Zhang  wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that
> > Istvan Toth(stoty)
> > has accepted the PMC's invitation to become a committer on the
> > project. We appreciate all
> > of Istvan Toth's generous contributions thus far and look forward to
> > his continued involvement.
> >
> > Congratulations and welcome, Istvan Toth!
> >
> > 我很高兴代表 Apache HBase PMC 宣布 Istvan Toth 已接受我们的邀请，成
> > 为 Apache HBase 项目的 Committer。感谢 Istvan Toth 一直以来为 HBase 项目
> > 做出的贡献，并期待他在未来继续承担更多的责任。
> >
> > 欢迎 Istvan Toth！
> >
>

Re: [ANNOUNCE] New HBase committer Nihal Jain

[Wei-Chiu Chuang]

Congrats!

On Wed, May 3, 2023 at 5:42 AM rajeshb...@apache.org <
chrajeshbab...@gmail.com> wrote:

> Congratulations Nihal!!!
>
>
> Thanks,
> Rajeshbabu.
>
> On Wed, May 3, 2023, 5:42 PM Nick Dimiduk  wrote:
>
> > Hello!
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal
> Jain
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Nihal's generous contributions thus far and look
> forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Nihal Jain!
> >
> > Thanks,
> > Nick
> >
>

Re: [ANNOUNCE] New HBase committer Nihal Jain

[Wei-Chiu Chuang]

Congrats!

On Wed, May 3, 2023 at 5:42 AM rajeshb...@apache.org <
chrajeshbab...@gmail.com> wrote:

> Congratulations Nihal!!!
>
>
> Thanks,
> Rajeshbabu.
>
> On Wed, May 3, 2023, 5:42 PM Nick Dimiduk  wrote:
>
> > Hello!
> >
> > On behalf of the Apache HBase PMC, I am pleased to announce that Nihal
> Jain
> > has accepted the PMC's invitation to become a committer on the project.
> We
> > appreciate all of Nihal's generous contributions thus far and look
> forward
> > to his continued involvement.
> >
> > Congratulations and welcome, Nihal Jain!
> >
> > Thanks,
> > Nick
> >
>

Re: [ANNOUNCE] New HBase committer Rushabh Shah

[Wei-Chiu Chuang]

Congrats!

On Thu, Dec 15, 2022 at 7:50 AM Geoffrey Jacoby  wrote:

> Congratulations, Rushabh!
>
> Geoffrey
>
> On Thu, Dec 15, 2022 at 10:26 AM Tak Lon (Stephen) Wu 
> wrote:
>
> > Congratulations, and welcome!
> >
> > -Stephen
> >
> > On Thu, Dec 15, 2022 at 5:39 AM Viraj Jasani  wrote:
> >
> > > Very well deserved! Congratulations and Welcome, Rushabh!!
> > >
> > >
> > > On Wed, Dec 14, 2022 at 10:57 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > > Rushabh Shah(shahrs87)
> > > > has accepted the PMC's invitation to become a committer on the
> > > > project. We appreciate all
> > > > of Rushabh's generous contributions thus far and look forward to his
> > > > continued involvement.
> > > >
> > > > Congratulations and welcome, Rushabh Shah!
> > > >
> > > > 我很高兴代表 Apache HBase PMC 宣布 Rushabh Shah 已接受我们的邀请，成
> > > > 为 Apache HBase 项目的 Committer。感谢 Rushabh Shah 一直以来为 HBase 项目
> > > > 做出的贡献，并期待他在未来继续承担更多的责任。
> > > >
> > > > 欢迎 Rushabh Shah！
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase committer Rushabh Shah

[Wei-Chiu Chuang]

Congrats!

On Thu, Dec 15, 2022 at 7:50 AM Geoffrey Jacoby  wrote:

> Congratulations, Rushabh!
>
> Geoffrey
>
> On Thu, Dec 15, 2022 at 10:26 AM Tak Lon (Stephen) Wu 
> wrote:
>
> > Congratulations, and welcome!
> >
> > -Stephen
> >
> > On Thu, Dec 15, 2022 at 5:39 AM Viraj Jasani  wrote:
> >
> > > Very well deserved! Congratulations and Welcome, Rushabh!!
> > >
> > >
> > > On Wed, Dec 14, 2022 at 10:57 PM 张铎(Duo Zhang) 
> > > wrote:
> > >
> > > > On behalf of the Apache HBase PMC, I am pleased to announce that
> > > > Rushabh Shah(shahrs87)
> > > > has accepted the PMC's invitation to become a committer on the
> > > > project. We appreciate all
> > > > of Rushabh's generous contributions thus far and look forward to his
> > > > continued involvement.
> > > >
> > > > Congratulations and welcome, Rushabh Shah!
> > > >
> > > > 我很高兴代表 Apache HBase PMC 宣布 Rushabh Shah 已接受我们的邀请，成
> > > > 为 Apache HBase 项目的 Committer。感谢 Rushabh Shah 一直以来为 HBase 项目
> > > > 做出的贡献，并期待他在未来继续承担更多的责任。
> > > >
> > > > 欢迎 Rushabh Shah！
> > > >
> > >
> >
>

Re: [ANNOUNCE] New HBase Committer Liangjun He

[Wei-Chiu Chuang]

Congrats!

On Sun, Dec 4, 2022 at 5:33 AM 宾莉金（binlijin）  wrote:

> Congratulations!
>
> 张铎(Duo Zhang)  于2022年12月3日周六 22:28写道：
>
> > Congratulations!
> >
> > Yu Li  于2022年12月3日周六 21:51写道：
> > >
> > > Hi All,
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Liangjun
> > > He (heliangjun) has accepted the PMC's invitation to become a committer
> > on
> > > the project. We appreciate all of Liangjun's generous contributions
> thus
> > > far and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Liangjun!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase
> 项目的
> > > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎良均！
> > >
> > > Best Regards,
> > > Yu
> > > --
> > > Best Regards,
> > > Yu
> >
>
>
> --
> *Best Regards,*
>  lijin bin
>

Re: [ANNOUNCE] New HBase Committer Liangjun He

[Wei-Chiu Chuang]

Congrats!

On Sun, Dec 4, 2022 at 5:33 AM 宾莉金（binlijin）  wrote:

> Congratulations!
>
> 张铎(Duo Zhang)  于2022年12月3日周六 22:28写道：
>
> > Congratulations!
> >
> > Yu Li  于2022年12月3日周六 21:51写道：
> > >
> > > Hi All,
> > >
> > > On behalf of the Apache HBase PMC, I am pleased to announce that
> Liangjun
> > > He (heliangjun) has accepted the PMC's invitation to become a committer
> > on
> > > the project. We appreciate all of Liangjun's generous contributions
> thus
> > > far and look forward to his continued involvement.
> > >
> > > Congratulations and welcome, Liangjun!
> > >
> > > 我很高兴代表 Apache HBase PMC 宣布 Liangjun He (何良均) 已接受我们的邀请，成为 Apache HBase
> 项目的
> > > Committer。感谢何良均一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> > >
> > > 欢迎良均！
> > >
> > > Best Regards,
> > > Yu
> > > --
> > > Best Regards,
> > > Yu
> >
>
>
> --
> *Best Regards,*
>  lijin bin
>

Re: HBase 2.4.x + Spark 3.3

[Wei-Chiu Chuang]

This looks similar to what we are seeing with Ozone running w/ Spark 3.3
HDDS-6926 

My guess is Spark 3.3 switched to the shaded Hadoop client jar, where
protobuf message classes are shaded, and somehow that breaks applications
using old Hadoop classes.

If you can share some stack traces that would be great.


On Mon, Oct 24, 2022 at 11:26 AM Lars Francke 
wrote:

> Hi Andrew,
>
> okay, we'll try that and will report back when/if we get it working.
>
> Cheers,
> Lars
>
> On Thu, Oct 20, 2022 at 2:29 AM Andrew Purtell 
> wrote:
>
> > No, that is insufficient. HBase must be recompiled against Hadoop 3 first
> >
> > cd /path/to/hbase
> > mvn clean install assembly:single -DskipTests -Dhadoop.profile=3.0
> > -Dhadoop-three.version=XXX
> >
> > Then once the results are in your local maven cache or nexus instance,
> you
> > can compile Spark as indicated.
> >
> >
> > On Tue, Oct 18, 2022 at 11:39 PM Lars Francke 
> > wrote:
> >
> > > Hi Andrew,
> > >
> > > thanks for the reply.
> > > I should have been more specific: We only tried to compile the "client"
> > > part that's used in Spark itself and we used the proper versions
> > >
> > > mvn -Dspark.version=XXX -Dscala.version=XXX -Dhadoop-three.version=XXX
> > > -Dscala.binary.version=XXX -Dhbase.version=XXX clean package
> > >
> > > I assume that should pull in the correct dependencies but I have to
> admit
> > > that I didn't check, took it straight from the readme.
> > > We wanted to try the server bit for the RegionServers afterwards but
> > didn't
> > > even get to it yet.
> > >
> > > We have this on our radar though and might try to work through those
> > issues
> > > at some point.
> > > If we get started on that I'll ping the list.
> > >
> > > Cheers,
> > > Lars
> > >
> > > On Wed, Oct 19, 2022 at 1:41 AM Andrew Purtell 
> > > wrote:
> > >
> > > > Out of the box use is going to be problematic without recompiling
> HBase
> > > for
> > > > Hadoop 3. Spark 3.3 ships with Hadoop 3.3.2. Apache HBase 2.4.x (and
> > all
> > > > 2.x) releases are compiled against Hadoop 2. Link errors
> > (ClassNotFound,
> > > > NoClassDef, etc) I think are to be expected because the class
> > hierarchies
> > > > of various Hadoop things have been incompatibly changed in 3.x
> releases
> > > > relative to 2.x. This is not unreasonable. Semantic versioning
> suggests
> > > > breaking changes can be expected in a major version increment.
> > > >
> > > > Users probably need to do a holistic (or hermetic, if you prefer)
> build
> > > of
> > > > their bill of materials before testing it or certainly before
> shipping
> > > it.
> > > > Build your HBase for the version of Hadoop you are actually shipping
> it
> > > > with, as opposed to whatever the upstream project picks as a default
> > > build
> > > > target. They are called "convenience binaries" by the project and the
> > > > Foundation for a reason. Convenience may vary according to your
> > > > circumstances. When HBase finally ships builds compiled against
> Hadoop
> > 3
> > > by
> > > > default, anyone still using 2.x in production will face the same
> > problem
> > > > (in reverse). The Phoenix project also faces this issue for what it's
> > > > worth. Their readme and build instructions walk users through
> > rebuilding
> > > > HBase using -Dhadoop.profile=3.0 as a first step as well.
> > > >
> > > >
> > > > On Mon, Oct 17, 2022 at 1:52 PM Lars Francke  >
> > > > wrote:
> > > >
> > > > > Hi everyone,
> > > > >
> > > > > we've just recently tried getting the HBase Spark connector running
> > > > against
> > > > > Spark 3.3 and HBase 2.4.x and failed miserably. It was a mess of
> > Scala
> > > > and
> > > > > Java issues, classpath, NoClassDef etc.
> > > > >
> > > > > The trauma is too recent for me to dig up the details but if
> needed I
> > > can
> > > > > ;-)
> > > > >
> > > > > For now I'm just wondering if anyone has succeeded using this
> > > > combination?
> > > > >
> > > > > Cheers,
> > > > > Lars
> > > > >
> > > >
> > > >
> > > > --
> > > > Best regards,
> > > > Andrew
> > > >
> > > > Unrest, ignorance distilled, nihilistic imbeciles -
> > > > It's what we’ve earned
> > > > Welcome, apocalypse, what’s taken you so long?
> > > > Bring us the fitting end that we’ve been counting on
> > > >- A23, Welcome, Apocalypse
> > > >
> > >
> >
> >
> > --
> > Best regards,
> > Andrew
> >
> > Unrest, ignorance distilled, nihilistic imbeciles -
> > It's what we’ve earned
> > Welcome, apocalypse, what’s taken you so long?
> > Bring us the fitting end that we’ve been counting on
> >- A23, Welcome, Apocalypse
> >
>

Re: [ANNOUNCE] New HBase committer Yutong Xiao(肖禹同)

[Wei-Chiu Chuang]

Congrats! Yutong did a ton of work. I'm happy to see Yuton getting the
recognition!

On Fri, Mar 4, 2022 at 5:24 PM Nick Dimiduk  wrote:

> Congratulations Yutong, and thank you for all your contributions!
>
> On Wed, Mar 2, 2022 at 8:50 AM 张铎(Duo Zhang) 
> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Yutong
> > Xiao(YutSean) has accepted the PMC's invitation to become a committer on
> > the project. We appreciate all of Yutong's generous contributions thus
> far
> > and look forward to his continued involvement.
> >
> > Congratulations and welcome, Yutong Xiao!
> >
> > 我很高兴代表 Apache HBase PMC 宣布肖禹同已接受我们的邀请，成为 Apache HBase 项目的
> > Committer。感谢肖禹同一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> >
> > 欢迎肖禹同！
> >
>

Re: [ANNOUNCE] New HBase committer Yutong Xiao(肖禹同)

[Wei-Chiu Chuang]

Congrats! Yutong did a ton of work. I'm happy to see Yuton getting the
recognition!

On Fri, Mar 4, 2022 at 5:24 PM Nick Dimiduk  wrote:

> Congratulations Yutong, and thank you for all your contributions!
>
> On Wed, Mar 2, 2022 at 8:50 AM 张铎(Duo Zhang) 
> wrote:
>
> > On behalf of the Apache HBase PMC, I am pleased to announce that Yutong
> > Xiao(YutSean) has accepted the PMC's invitation to become a committer on
> > the project. We appreciate all of Yutong's generous contributions thus
> far
> > and look forward to his continued involvement.
> >
> > Congratulations and welcome, Yutong Xiao!
> >
> > 我很高兴代表 Apache HBase PMC 宣布肖禹同已接受我们的邀请，成为 Apache HBase 项目的
> > Committer。感谢肖禹同一直以来为 HBase 项目做出的贡献，并期待他在未来继续承担更多的责任。
> >
> > 欢迎肖禹同！
> >
>

Re: upgrade hadoop , keep hbase

[Wei-Chiu Chuang]

tl; dr: Pretty sure it won't run.

Looking at the Hadoop 3.3 support jira (HBASE-22953
), most of the work was
to whitelist additional dependencies. However, the biggest change was
inside the asyncwall HBASE-23833
 due to the
relocated protobuf classes. Netty and jetty version updates are mostly
likely to fail your installation as well.

We strive to make Hadoop follow the public compatibility guidelines. But
from time to time we have to make changes across minor release lines
(between 3.1.2 and 3.3.0) and also unfortunately HBase uses a number of
private Hadoop APIs which can change without notice.

(I think we can close HBASE-22953 as done at this point)

On Tue, Dec 21, 2021 at 10:02 AM Josh Elser  wrote:

> Hi Michael!
>
> I have to give you the "default" HBase, but defer you to the Hadoop
> community (or someone from there who is paying attention to this list)
> for the precise answer.
>
> The question you are asking is: does the binary compatibility of Hadoop
> jars changes between Hadoop 3.1.2 and 3.3.0. The short answer is "maybe".
>
> In general, within one Hadoop major version, the likelihood of a change
> which breaks binary compatibility [1] is relatively low. However, "low"
> is not "none". If you're doing this on a local environment or a dev
> system, you're probably OK. I would definitely recommend you recompile
> HBase if you're using it for a production system. You wouldn't want to
> be chasing a fix for this if it manifests in a subtle/strange manner.
>
> - Josh
>
> [1] https://docs.oracle.com/javase/specs/jls/se7/html/jls-13.html
>
> On 12/20/21 11:01 AM, Michael Wohlwend wrote:
> > Hello all,
> >
> > I have hadoop 3.1.2 running, with hbase 2.1.4
> >
> > If I want to update to hadoop 3.3.0,  is it sufficient to replace all
> the hadoop
> > 3.1.2 jars in the hbase folder tree with the 3.3.0 version?
> > Or do I have to make a new hbase build?
> >
> >
> > Thanks for answering
> >   Michael
> >
> >
> >
>

Re: [ANNOUNCE] New HBase committer Baiqiang Zhao

[Wei-Chiu Chuang]

great to hear that. Congrats!

On Sun, Jul 11, 2021 at 3:18 AM Nick Dimiduk  wrote:

> Hi everyone,
>
> On behalf of the Apache HBase PMC I am pleased to announce that Baiqiang
> Zhao has accepted the PMC's invitation to become a committer on the
> project!
>
> We appreciate all of the great contributions Baiqiang has made to
> the community thus far and we look forward to his continued involvement.
>
> Allow me to be the first to congratulate Baiqiang on his new role!
>
> Thanks,
> Nick
>

Re: Is HBase 2.X suitable for running on Hadoop 3.X?

[Wei-Chiu Chuang]

Filed HBASE-25771 <https://issues.apache.org/jira/browse/HBASE-25771> to
recommend Hadoop 3.x
Feel free to chime in.

On Fri, Apr 2, 2021 at 7:00 PM Wei-Chiu Chuang  wrote:

> i think it's time to remove that statement. We have lots of production
> users running HBase 2 on Hadoop 3 for several years now.
>
> On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang) 
> wrote:
>
>> According to the compatibility matrix, HBase 2.3.4 could work together
>> with
>> Hadoop 3.2.2.
>>
>> And if you just want to connec to HDFS 3.2.2, you could just use the pre
>> built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client in
>> HBase, you need to build the artifacts by your own.
>>
>> Thanks.
>>
>> hossein ahmadzadeh  于2021年3月31日周三 上午3:30写道：
>>
>> > <https://stackoverflow.com/posts/66877227/timeline>
>> >
>> > The HBase documentation <http://hbase.apache.org/book.html#hadoop>
>> noted
>> > that:
>> >
>> > Hadoop 2.x is faster and includes features, such as short-circuit reads
>> > (See Leveraging local data), which will help improve your HBase random
>> read
>> > profile. Hadoop 2.x also includes important bug fixes that will improve
>> > your overall HBase experience. HBase does not support running with
>> earlier
>> > versions of Hadoop. See the table below for requirements specific to
>> > different HBase versions. *Hadoop 3.x is still in early access releases
>> and
>> > has not yet been sufficiently tested by the HBase community for
>> production
>> > use cases.*
>> >
>> > But right next to this point in the compatibility table says that HBase
>> > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
>> > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
>> not?
>> >
>> > ‌Best Regards.
>> >
>>
>

Re: [ANNOUNCE] New HBase PMC Huaxiang Sun

[Wei-Chiu Chuang]

Congrats!

On Tue, Apr 13, 2021 at 3:39 PM Viraj Jasani  wrote:

> On behalf of the Apache HBase PMC I am pleased to announce that Huaxiang
> Sun has accepted our invitation to become a PMC member on the HBase
> project. We appreciate Huaxiang stepping up to take more responsibility for
> the project.
>
> Congratulations and welcome, Huaxiang!
>

Re: Is HBase 2.X suitable for running on Hadoop 3.X?

[Wei-Chiu Chuang]

i think it's time to remove that statement. We have lots of production
users running HBase 2 on Hadoop 3 for several years now.

On Fri, Apr 2, 2021 at 6:32 PM 张铎(Duo Zhang)  wrote:

> According to the compatibility matrix, HBase 2.3.4 could work together with
> Hadoop 3.2.2.
>
> And if you just want to connec to HDFS 3.2.2, you could just use the pre
> built artifacts for HBase 2.3.4. If you want to use Hadoop 3.2.2 client in
> HBase, you need to build the artifacts by your own.
>
> Thanks.
>
> hossein ahmadzadeh  于2021年3月31日周三 上午3:30写道：
>
> > 
> >
> > The HBase documentation  noted
> > that:
> >
> > Hadoop 2.x is faster and includes features, such as short-circuit reads
> > (See Leveraging local data), which will help improve your HBase random
> read
> > profile. Hadoop 2.x also includes important bug fixes that will improve
> > your overall HBase experience. HBase does not support running with
> earlier
> > versions of Hadoop. See the table below for requirements specific to
> > different HBase versions. *Hadoop 3.x is still in early access releases
> and
> > has not yet been sufficiently tested by the HBase community for
> production
> > use cases.*
> >
> > But right next to this point in the compatibility table says that HBase
> > 2.3.4 is fully functional with Hadoop 3.2.2. So I got confused about
> > whether We can use HBase 2.3.4 in production alongside Hadoop 3.2.2 or
> not?
> >
> > ‌Best Regards.
> >
>

Re: 求助HBase社区关于HBase2.1.0 主备复制的一系列异常现象

[Wei-Chiu Chuang]

圖片似乎還是無法顯示 煩請重發一次

On Wed, Dec 30, 2020 at 7:05 PM sudo rm -rf /* <2326130...@qq.com> wrote:

> hi
>
> 刚刚那封邮件图片显示可能有问题，重发一次。
>
> 我们线上HBase为主备两个集群，主备集群开启replication，模式为主主模式，即可以互相同步数据。HBase版本
> hbase2.1.0-cdh-6.3.2
>
> 目前多次遇到以下异常现象。
>
> 第一：replication 积压，replication队列持续堆积不会减少，目前存在九月份的WAL Log还在replication队列中。如下图
>
>
>
>
>
> 图中积压的replication数据时间戳显示九月份
>
> 第二个异常，WALs oldWals目录异常庞大
>
> 主集群
>
>
> 备集群
>
>
> Replication相关线程的信息如下图
>
> 希望得到社区的帮助，帮忙提供问题分析的思路，感谢感谢。目前不知道是replication的BUG还是我们的使用姿势不对
>
>
>

Re: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC

[Wei-Chiu Chuang]

Congratulations!

On Tue, Oct 6, 2020 at 5:18 AM 张铎(Duo Zhang)  wrote:

> Congratulations!
>
> zheng wang <18031...@qq.com> 于2020年10月6日周二 下午8:00写道：
>
> > CongratulationsViraj!
> >
> >
> >
> >
> > ---Original---
> > From: "Andrew Purtell" > Date: Tue, Oct 6, 2020 00:58 AM
> > To: "Hbase-User" ;
> > Subject: [ANNOUNCE] Please welcome Viraj Jasani to the Apache HBase PMC
> >
> >
> > On behalf of the Apache HBase PMC I am pleased to announce that
> > Viraj Jasani has accepted our invitation to become a PMC member on the
> > HBase project. We appreciate Viraj stepping up to take more
> > responsibility for the project.
> >
> > Please join me in welcoming Viraj to the HBase PMC!
> >
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or
> > PMC member, you can always drop a note to priv...@hbase.apache.org
> > to let us know.
> >
> > --
> > Best regards,
> > Andrew
>

Re: org.apache.hadoop.net.DNS exception on ipv6

[Wei-Chiu Chuang]

There is no ipv6 support in Hadoop.

But there was a feature branch in Hadoop to implement ipv6. It didn't merge
into trunk though.
https://issues.apache.org/jira/browse/HADOOP-11890

On Mon, Mar 16, 2020 at 8:55 AM Stack  wrote:

> On Fri, Mar 13, 2020 at 7:41 PM tomas tim  wrote:
>
> > hi,all
> > I start hbase in a vmware environment using following  config and fail.
> > the reason is org.apache.hadoop.net.DNS can not adapt IPv6 address, is
> this
> > a bug? what can I do except turn off ipv6 config?
> >
> > config:
> > 
> >   hbase.master.dns.interface
> >   ens33
> > 
> >
> > Hadoop doesn't support ipv6. HBase doesn't either.
> Thanks,
> S
>
>
>
>
> > exception:
> > java.lang.RuntimeException: Failed construction of Master: class
> > org.apache.hadoop.hbase.master.HMaster.
> > at
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2996)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.startMaster(HMasterCommandLine.java:236)
> > at
> >
> >
> org.apache.hadoop.hbase.master.HMasterCommandLine.run(HMasterCommandLine.java:140)
> > at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> > at
> >
> >
> org.apache.hadoop.hbase.util.ServerCommandLine.doMain(ServerCommandLine.java:149)
> > at org.apache.hadoop.hbase.master.HMaster.main(HMaster.java:3007)
> > Caused by: java.lang.RuntimeException: Failed to invoke
> DNS.getDefaultHost
> > via reflection
> > at org.apache.hadoop.hbase.util.DNS.getDefaultHost(DNS.java:67)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.getHostname(RSRpcServices.java:1329)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.(RSRpcServices.java:1228)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.RSRpcServices.(RSRpcServices.java:1206)
> > at
> >
> >
> org.apache.hadoop.hbase.master.MasterRpcServices.(MasterRpcServices.java:384)
> > at
> >
> org.apache.hadoop.hbase.master.HMaster.createRpcServices(HMaster.java:740)
> > at
> >
> >
> org.apache.hadoop.hbase.regionserver.HRegionServer.(HRegionServer.java:593)
> > at org.apache.hadoop.hbase.master.HMaster.(HMaster.java:513)
> > at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> > at
> >
> >
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> > at
> > org.apache.hadoop.hbase.master.HMaster.constructMaster(HMaster.java:2989)
> > ... 5 more
> > Caused by: java.lang.reflect.InvocationTargetException
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > at
> >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > at java.lang.reflect.Method.invoke(Method.java:498)
> > at org.apache.hadoop.hbase.util.DNS.getDefaultHost(DNS.java:64)
> > ... 17 more
> > Caused by: java.lang.ArrayIndexOutOfBoundsException: 3
> > at org.apache.hadoop.net.DNS.reverseDns(DNS.java:82)
> > at org.apache.hadoop.net.DNS.getHosts(DNS.java:253)
> > at org.apache.hadoop.net.DNS.getDefaultHost(DNS.java:366)
> > ... 22 more
> >
> >
> > Thanks,
> >
>

Re: [DISCUSS] Bump hadoop versions on up coming 2.3.0 and 3.0.0 releases

[Wei-Chiu Chuang]

+1 I'd like to encourage the community to update and reduce the Hadoop 2.x
presence as much as possible.

There is not an official DISCUSS/VOTE thread to EOL Hadoop 2.9 yet, even
though so far the feedback has been quite receptive to the idea.

Hadoop 2.10 is meant to be a "bridge release" for those who are not ready
to upgrade to Hadoop 3.x.
Given that the main contributors (LinkedIn, Microsoft and Verizon Media)
are on 2.10, bypassing 2.9 altogether, the Hadoop 2.9 is not going to get
much attention. Looking at git history, branch-2.9's got just 13 commits.



On Wed, Mar 11, 2020 at 5:35 PM 张铎(Duo Zhang)  wrote:

> Hadoop now has a wiki page to show the EOL releases lines
>
>
> https://cwiki.apache.org/confluence/display/HADOOP/EOL+(End-of-life)+Release+Branches
>
>
> 2.8.x is finally dead, so we'd better at least upgrade to 2.9.x in newer
> release lines.
>
> And in this announcement
>
>
> https://lists.apache.org/thread.html/r348f7bc93a522f05b7cce78a911854d128a6b1b8bd8124bad4d06ce6%40%3Cuser.hadoop.apache.org%3E
>
>
> They even say you'd better upgrade to 2.10+.
>
> For me, I prefer we just bump the hadoop version of 2.3.0 directly to
> 2.10.0, as 2.9.x is almost dead too. And since we still support hadoop
> 2.9.x on 2.2.x release line, which is the current stable release line,
> which should be fine for users.
>
> And for master branch, I suggest we just drop all the support for hadoop
> 2.x and bump hadoop to 3.1.x directly.
>
> Thoughts? Thanks.
>

Re: Hbase trace

[Wei-Chiu Chuang]

As far as I know the HTrace server is not supported by CDH6. (It didn't
work the last time I tried)

On Sun, Jan 19, 2020 at 8:31 PM qq <471237...@qq.com> wrote:

> Oh . Thanks very much. I use 2.1.0-cdh6.3.1 version. And how can I also
> trace hbase like Htrace on the version ?
>
> Alex Fu
>
>
> > 2020年1月19日 21:13，张铎(Duo Zhang)  写道：
> >
> > Which version do you use? For 2.x, the htrace system is broken.
> >
> > qq <471237...@qq.com > 于2020年1月19日周日 上午10:51写道：
> > Hi all.
> >
> >When I use Hbase Htrace as document, but I can’t get any tracing
> information ? Why ? Thanks very much.
> >
> >
> >
> > 
> >   hbase.trace.spanreceiver.classes
> >   org.apache.htrace.core.LocalFileSpanReceiver
> > 
> > 
> >   hbase.htrace.local-file-span-receiver.path
> >   /var/log/hbase/htrace.out
> > 
> >
> >
> >
> >
> >
> > Up is my test image.
> >
> >But I can’t find file which is /var/log/hbase/htrace.out on my hbase
> cluster nodes.
> >
> >
> > Alex Fu
> >
>
>

Re: [DISCUSS] Bump hadoop versions

[Wei-Chiu Chuang]

With my Hadoop hat's on, we have not yet officially declared Hadoop 2.8
EOL. I think the 2.8 download missing from the web page is just a mistake.

That being said, some of the biggest Hadoop users (LinkedIn, Yahoo,
Microsoft) that I am aware of are moving up from 2.7/2.8 to 2.10, and that
2.8.5 (the last version in the 2.8 line) was released in Sep 2018, more
than a year ago. It doesn't look like the community has the desire to
continue the 2.8 line.

I think it is a little extreme to remove hadoop2 profile, given that Hadoop
2.9 and 2.10 are still active and I expect Hadoop 2 to stay around for at
least 2 years out.

On Thu, Dec 26, 2019 at 8:41 AM 张铎(Duo Zhang)  wrote:

> Hadoop 2.8.x has been removed from the download page of hadoop so I think
> it is time to bump the hadoop dependency to 2.9.x, on master an branch-2.
>
> And the hadoop community is going to make 2.10.x the last minor release
> line for 2.x
>
>
> https://lists.apache.org/thread.html/cab84265d632b90d66dcd1ad957a7439a2c76a987c7e62feafb4812e%40%3Ccommon-dev.hadoop.apache.org%3E
>
>
> I think this is a sign that the community is moving forward to 3.x. So I
> propose we make the master branch hadoop3 only, This requires changing the
> pom a bit to active hadoop3 profile by default and remove the hadoop2
> profile.
>
> Thoughts? Thanks.
>

Re: Does Hedged-read usually happen in real workloads?

[Wei-Chiu Chuang]

On Mon, Aug 19, 2019 at 5:30 AM Daegyu Han  wrote:

> Hi all,
>
> To my understand, Hedged-Read is a technique for copying IO to a replica
> when pread () is delayed to mitigate IO latency such as speculative
> execution in MapReduce.
>
> - Does Hedged-read really happen where it uses hbase databases?
>
There was a pretty good talk at HBaseCon 2018
https://www.youtube.com/watch?v=l6S-Vbs9WsU=PLdOhKvENNn6WH71iHXRNbEd1pTN2xY6P3=9
It talked about HBase Read Replica, but at the same time, also touched upon
hedged read.

Additionally, I am aware this is enabled for HBase and Impala among my
customers.
It's not clear to me what impact does it bring to Impala, but Impala is
quite sensitive to slow HDFS as well.

> - Does hedged-read often occur because of a bottleneck in HDFS or a
> bottleneck in the storage itself?
> - What is the best number to set the timeout and thread for Hedged-Read?
>
There are some configurations mentioned in the talk above. Check it out.

>
> Thank you
> ᐧ
>

Re: [ANNOUNCE] New Committer: Wellington Chevreuil

[Wei-Chiu Chuang]

Yay!! Congrats for the achievement!

On Fri, Jun 7, 2019 at 12:56 PM Andrew Purtell 
wrote:

> Congratulations and welcome, Wellington.
>
> > On Jun 7, 2019, at 3:11 AM, Peter Somogyi  wrote:
> >
> > On behalf of the HBase PMC, I'm pleased to announce that Wellington
> > Chevreuil has accepted our invitation to become an HBase committer.
> >
> > Thanks for all your hard work Wellington; we look forward to more
> > contributions!
> >
> > Please join me in extending congratulations to Wellington!
>

Re: Disk hot swap for data node while hbase use short-circuit

[Wei-Chiu Chuang]

I think i found a similar bug report that matches your symptom: HDFS-12204
<https://issues.apache.org/jira/browse/HDFS-12204> (Dfsclient Do not close
file descriptor when using shortcircuit)

On Wed, May 29, 2019 at 11:37 PM Kang Minwoo 
wrote:

> I think these file opened for reads. because that block is finalized.
>
> ---
> ls -al /proc/regionserver_pid/fd
> 902 -> /data_path/current/finalized/~/blk_1 (deleted)
> 946 -> /data_path/current/finalized/~/blk_2 (deleted)
> 947 -> /data_path/current/finalized/~/blk_3.meta (deleted)
> ---
>
> I think it is not an HBase bug. This is because DFSClient checks stale fd
> when the fetch method invoked.
>
> Best regards,
> Minwoo Kang
>
> ____
> 보낸 사람: Wei-Chiu Chuang 
> 보낸 날짜: 2019년 5월 29일 수요일 20:51
> 받는 사람: user@hbase.apache.org
> 제목: Re: Disk hot swap for data node while hbase use short-circuit
>
> Do you have a list of files that was being opened? I'd like to know if
> those are files opened for writes or for reads.
>
> If you are on the more recent version of Hadoop (2.8.0 and above),
> there's a HDFS command to interrupt ongoing writes to DataNodes (HDFS-9945
> <https://issues.apache.org/jira/browse/HDFS-9945>)
>
>
> https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin
> hdfs dfsadmin -evictWriters
>
> Looking at HDFS hotswap implementation, it looks like DataNode doesn't
> interrupt writers when a volume is removed. That sounds like a bug.
>
> On Tue, May 28, 2019 at 9:39 PM Kang Minwoo 
> wrote:
>
> > Hello, Users.
> >
> > I use JBOD for data node. Some times the disk in the data node has a
> > problem.
> >
> > The first time, I shut down all instance include data node and region
> > server in the machine that has a disk problem.
> > But It is not a good solution. So I improve the process.
> >
> > When I detect disk problem in the server. I just perform disk hot swap.
> >
> > But System administrator complains of some FD that still open so they
> > cannot remove the disk.
> > Regionserver has an FD, I use short circuit reads feature. (HBase version
> > 1.2.9)
> >
> > When we first met this issue, we force unmount disk and remount.
> > But after this process, kernel report error[1].
> >
> > So we avoid this issue. purge stale FD.
> >
> > I think this issue is common.
> > I want to know how hbase-users deal with this issue.
> >
> > Thank you very much for sharing your experience.
> >
> > Best regards,
> > Minwoo Kang
> >
> > [1]:
> >
> https://www.thegeekdiary.com/xfs_log_force-error-5-returned-xfs-error-centos-rhel-7/
> >
>

Re: Disk hot swap for data node while hbase use short-circuit

[Wei-Chiu Chuang]

Do you have a list of files that was being opened? I'd like to know if
those are files opened for writes or for reads.

If you are on the more recent version of Hadoop (2.8.0 and above),
there's a HDFS command to interrupt ongoing writes to DataNodes (HDFS-9945
)

https://hadoop.apache.org/docs/r2.8.5/hadoop-project-dist/hadoop-hdfs/HDFSCommands.html#dfsadmin
hdfs dfsadmin -evictWriters

Looking at HDFS hotswap implementation, it looks like DataNode doesn't
interrupt writers when a volume is removed. That sounds like a bug.

On Tue, May 28, 2019 at 9:39 PM Kang Minwoo  wrote:

> Hello, Users.
>
> I use JBOD for data node. Some times the disk in the data node has a
> problem.
>
> The first time, I shut down all instance include data node and region
> server in the machine that has a disk problem.
> But It is not a good solution. So I improve the process.
>
> When I detect disk problem in the server. I just perform disk hot swap.
>
> But System administrator complains of some FD that still open so they
> cannot remove the disk.
> Regionserver has an FD, I use short circuit reads feature. (HBase version
> 1.2.9)
>
> When we first met this issue, we force unmount disk and remount.
> But after this process, kernel report error[1].
>
> So we avoid this issue. purge stale FD.
>
> I think this issue is common.
> I want to know how hbase-users deal with this issue.
>
> Thank you very much for sharing your experience.
>
> Best regards,
> Minwoo Kang
>
> [1]:
> https://www.thegeekdiary.com/xfs_log_force-error-5-returned-xfs-error-centos-rhel-7/
>

Re: HBase top activity

[Wei-Chiu Chuang]

FWIW, Andrew proposed a htop utility:
https://issues.apache.org/jira/browse/HBASE-11062

On Wed, Jan 9, 2019 at 4:36 AM Wellington Chevreuil <
wellington.chevre...@gmail.com> wrote:

> I don't think it is possible to track individual scans resources usage,
> current available JMX metrics are an aggregation of all request types
> (scanNext, append, get, delete) per regions level at individual RSes, and
> these are basically counts on the number of those requests. Resources usage
> are measured at RS process level, so that also adds a challenge to account
> those for individual scans. Sure, you could break those down by looking at
> each thread usage with some monitoring tool such as VisualVM, but then you
> would still need to map each RPC handler to an specific scan, and depending
> on how large the result, a single scan would be addressed by different
> handlers every time it calls "next" fetch another batch of results it's
> iterating through.
>
> Em seg, 7 de jan de 2019 às 14:27, shicheng31...@gmail.com <
> shicheng31...@gmail.com> escreveu:
>
> > Maybe  coprocessor can  help you in some way.
> >
> >
> >
> > shicheng31...@gmail.com
> >
> > From: Meirav Malka
> > Date: 2019-01-07 19:22
> > To: Hbase User Group; dev
> > Subject: HBase top activity
> > Hi,
> >
> > Does anyone know i there's a way to get the current operations running
> in a
> > HBase cluster?
> > We want to be able to detect the running scans, their execution time,
> > number of executions,cpu, memory and io.(as can be found in Oracle
> > database)
> > Is there any tracking of this data in HBase?
> >
> > Thanks
> >
>

Re: Welcome Chia-Ping Tsai to the HBase PMC

[Wei-Chiu Chuang]

My sincere congratulations!

On Fri, Sep 29, 2017 at 3:22 PM, Ted Yu  wrote:

> Congratulations, Chia-Ping.
>
> On Fri, Sep 29, 2017 at 3:19 PM, Misty Stanley-Jones 
> wrote:
>
> > The HBase PMC is delighted to announce that Chia-Ping Tsai has agreed to
> > join
> > the HBase PMC, and help to make the project run smoothly. Chia-Ping
> became
> > an
> > HBase committer over 6 months ago, based on long-running participate in
> the
> > HBase project, a consistent record of resolving HBase issues, and
> > contributions
> > to testing and performance.
> >
> > Thank you for stepping up to serve, Chia-Ping!
> >
> > As a reminder, if anyone would like to nominate another person as a
> > committer or PMC member, even if you are not currently a committer or PMC
> > member, you can always drop a note to priv...@hbase.apache.org to let us
> > know!
> >
> > Thanks,
> > Misty (on behalf of the HBase PMC)
> >
>



-- 
A very happy Clouderan

Re: Please congratulate our new PMC Chair Misty Stanley-Jones

[Wei-Chiu Chuang]

Congrats! Misty!!

On Fri, Sep 22, 2017 at 7:50 AM, Jimmy Xiang  wrote:

> Congrats! Misty!!
>
> On Fri, Sep 22, 2017 at 7:16 AM, Pankaj kr  wrote:
>
> > Congratulations Misty..!! :)
> >
> >
> > -Pankaj-
> >
> >
> > -Original Message-
> > From: Andrew Purtell [mailto:apurt...@apache.org]
> > Sent: Friday, September 22, 2017 3:08 AM
> > To: d...@hbase.apache.org; user@hbase.apache.org
> > Subject: Please congratulate our new PMC Chair Misty Stanley-Jones
> >
> > At today's meeting of the Board, Special Resolution B changing the HBase
> > project Chair to Misty Stanley-Jones was passed unanimously.
> >
> > Please join me in congratulating Misty on her new role!
> >
> > ​(If you need any help or advice please don't hesitate to ping me, Misty,
> > but I suspect you'll do just fine and won't need it.)​
> >
> >
> > --
> > Best regards,
> > Andrew
> >
>



-- 
A very happy Hadoop contributor