[jira] [Created] (HBASE-26249) Request split and compact after

2021-09-01 Thread Xiaolin Ha (Jira) 
Xiaolin Ha created HBASE-26249:
--

 Summary: Request split and compact after
 Key: HBASE-26249
 URL: https://issues.apache.org/jira/browse/HBASE-26249
 Project: HBase
  Issue Type: Improvement
Reporter: Xiaolin Ha






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Merge feature branch HBASE-25853 into branch-2

2021-09-01 Thread Duo Zhang 
It is https://issues.apache.org/jira/browse/HADOOP-15566.

If anyone is interested, please help here. I do not have much time to add
this support to hadoop, as I still owe the hadoop community a big work on
the migration to log4j2...

Thanks.

张铎(Duo Zhang)  于2021年9月2日周四 上午7:31写道：

> Hadoop is not ready yet.
>
> Wei-Chiu Chuang should have opened an issue for it but no big progress yet.
>
> On a mobile device, let me report back later when I find the issue for
> Hadoop.
>
> Thanks.
>
> Nick Dimiduk 于2021年9月2日 周四02:46写道：
>
>> This is great information. I would feel better about it if we had
>> confirmation that everything is working as expected on branch-2. The reason
>> for this is that master and branch-2 have drifted quite a bit, and the
>> instrumentation points are quite sensitive to internal details of the code.
>> Thus, I don’t assume that behavior demonstrated on master is necessarily
>> applicable to branch-2.
>>
>> Since no one seems to have demonstrated on the feature branch what I’m
>> hoping for, let me spend some time with the feature branch and/or branch-2
>> after merge. I think it’s important that we know this works well, as it is
>> a headline feature of 2.5. (If anyone else is really keen to do this
>> demonstration, please let us know!)
>>
>> Related: what’s the status is tracing into HDFS and beyond? Should I
>> expect to see spans all the way down to HDFS disk access? What about kernel
>> system calls?
>>
>> Thanks,
>> Nick
>>
>> On Mon, Aug 30, 2021 at 20:30 张铎(Duo Zhang) 
>> wrote:
>>
>>> Before I left Xiaomi, I’ve already set up a testing cluster and
>>> collected the tracing data to Apache Skywalking and show it.
>>>
>>> There are some screen shots, I found them in the private slack channel
>>> between Stack and me.
>>>
>>>
>>>
>>> Tak Lon (Stephen) Wu 于2021年8月31日 周二10:09写道：
>>>
 Hey Nick, IMO you're good, and we should move your concern to another
 discussion topic.

 Let me reply here first and see if the created JIRA
 https://issues.apache.org/jira/browse/HBASE-26241 (could be a blocker
 for hbase-2.5.0 release) meets your exception.

 If I read your message correctly, even if we have the instruction on
 how to enable opentelemetery for writing trace in HBase's log [1]
 (should have tested in HBASE-25658 [2]) as well as we have the
 interface for other telemetry sinks [3], you would like to see the
 following before releasing 2.5.0 if we're merging HBASE-25853 into
 branch-2?

 a. Someone (either the RM or me) can spend a bit on testing end-2-end
 with a graph as result in the JIRA (or hbase book html) that this
 tracing can be rendered at the sink of an external telemetry system
 e.g. Jaeger [4] or zipkin [5].
 b. For a `service call`, e.g. via a put/get call, how does the trace
 message look like in logs or via from the UI?

 [1] https://hbase.apache.org/book.html#tracing
 [2] https://issues.apache.org/jira/browse/HBASE-25658 has the basic
 benchmark
 [3]
 https://danw1ld.medium.com/observability-for-front-end-web-clients-with-opentelemetry-and-jaeger-in-5-minutes-343f719fbf5a
 [4] Jaeger, https://github.com/jaegertracing/jaeger
 [5] zipkin, https://github.com/openzipkin/zipkin

 Thanks,
 Stephen


 On Mon, Aug 30, 2021 at 5:41 PM Nick Dimiduk 
 wrote:
 >
 > +0
 >
 > Recent PR test failures are unfortunate, but look unrelated. It's
 > unfortunate, because there's enough change here that I think we'd all
 be a
 > lot more comfortable with better confidence that the changeset has not
 > introduced instabilities.
 >
 > I think we're pretty close to cutting a 2.5.0 from this branch. I've
 not
 > seen anyone report their success at integrating these changes into a
 > telemetry system and showing end-to-end tracing of a service call. As
 I
 > recall from the last 2.5 discuss thread, we wanted this to be the
 banner
 > feature for the minor release. Since we don't have this end-to-end
 > confirmation, do we have someone who's volunteered to demonstrate that
 > final proof-of-integration in a timely manner? Perhaps that someone
 is keen
 > to be release manager for 2.5.
 >
 > I didn't see a discuss thread talking about what open items might be
 left
 > before merge, so I raise my question here. Pardon me if these comments
 > should have landed elsewhere.
 >
 > Thanks,
 > Nick
 >
 > On Sun, Aug 29, 2021 at 8:22 PM Tak Lon (Stephen) Wu <
 tak...@apache.org>
 > wrote:
 >
 > > Hi everyone,
 > >
 > > I'm writing this request and propose a merge of HBASE-25853
 > >  "Backport
 HBASE-22120
 > > Replace HTrace with OpenTelemetry" to branch-2. The goal is to
 remove
 > > HTrace and uses OpenTelemetry also in branch-2.5.0+.
 > >
 > >

[jira] [Resolved] (HBASE-26147) Add dry run mode to hbase balancer

2021-09-01 Thread Josh Elser (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26147?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Josh Elser resolved HBASE-26147.

Resolution: Fixed

Thanks for the excellent work, Bryan. This really came together to be a great 
change. I took the liberty of writing some release notes – please feel free to 
update them as you see fit.

Thanks Duo, Nick, and everyone else who helped out in reviews.

My apologies that I botched application of the branch-2 PR (putting my own 
email address instead of Bryan's). I reverted my original commit and re-applied 
it with correct metadata. Sorry for sullying the commit log. 

> Add dry run mode to hbase balancer
> --
>
> Key: HBASE-26147
> URL: https://issues.apache.org/jira/browse/HBASE-26147
> Project: HBase
>  Issue Type: Improvement
>  Components: Balancer, master
>Reporter: Bryan Beaudreault
>Assignee: Bryan Beaudreault
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2
>
>
> It's often rather hard to know how the cost function changes you're making 
> will affect the balance of the cluster, and currently the only way to know is 
> to run it. If the cost decisions are not good, you may have just moved many 
> regions towards a non-ideal balance. Region moves themselves are not free for 
> clients, and the resulting balance may cause a regression.
> We should add a mode to the balancer so that it can be invoked without 
> actually executing any plans. This will allow an administrator to iterate on 
> their cost functions and used the balancer's logging to see how their changes 
> would affect the cluster. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Merge feature branch HBASE-25853 into branch-2

2021-09-01 Thread Duo Zhang 
Hadoop is not ready yet.

Wei-Chiu Chuang should have opened an issue for it but no big progress yet.

On a mobile device, let me report back later when I find the issue for
Hadoop.

Thanks.

Nick Dimiduk 于2021年9月2日 周四02:46写道：

> This is great information. I would feel better about it if we had
> confirmation that everything is working as expected on branch-2. The reason
> for this is that master and branch-2 have drifted quite a bit, and the
> instrumentation points are quite sensitive to internal details of the code.
> Thus, I don’t assume that behavior demonstrated on master is necessarily
> applicable to branch-2.
>
> Since no one seems to have demonstrated on the feature branch what I’m
> hoping for, let me spend some time with the feature branch and/or branch-2
> after merge. I think it’s important that we know this works well, as it is
> a headline feature of 2.5. (If anyone else is really keen to do this
> demonstration, please let us know!)
>
> Related: what’s the status is tracing into HDFS and beyond? Should I
> expect to see spans all the way down to HDFS disk access? What about kernel
> system calls?
>
> Thanks,
> Nick
>
> On Mon, Aug 30, 2021 at 20:30 张铎(Duo Zhang)  wrote:
>
>> Before I left Xiaomi, I’ve already set up a testing cluster and collected
>> the tracing data to Apache Skywalking and show it.
>>
>> There are some screen shots, I found them in the private slack channel
>> between Stack and me.
>>
>>
>>
>> Tak Lon (Stephen) Wu 于2021年8月31日 周二10:09写道：
>>
>>> Hey Nick, IMO you're good, and we should move your concern to another
>>> discussion topic.
>>>
>>> Let me reply here first and see if the created JIRA
>>> https://issues.apache.org/jira/browse/HBASE-26241 (could be a blocker
>>> for hbase-2.5.0 release) meets your exception.
>>>
>>> If I read your message correctly, even if we have the instruction on
>>> how to enable opentelemetery for writing trace in HBase's log [1]
>>> (should have tested in HBASE-25658 [2]) as well as we have the
>>> interface for other telemetry sinks [3], you would like to see the
>>> following before releasing 2.5.0 if we're merging HBASE-25853 into
>>> branch-2?
>>>
>>> a. Someone (either the RM or me) can spend a bit on testing end-2-end
>>> with a graph as result in the JIRA (or hbase book html) that this
>>> tracing can be rendered at the sink of an external telemetry system
>>> e.g. Jaeger [4] or zipkin [5].
>>> b. For a `service call`, e.g. via a put/get call, how does the trace
>>> message look like in logs or via from the UI?
>>>
>>> [1] https://hbase.apache.org/book.html#tracing
>>> [2] https://issues.apache.org/jira/browse/HBASE-25658 has the basic
>>> benchmark
>>> [3]
>>> https://danw1ld.medium.com/observability-for-front-end-web-clients-with-opentelemetry-and-jaeger-in-5-minutes-343f719fbf5a
>>> [4] Jaeger, https://github.com/jaegertracing/jaeger
>>> [5] zipkin, https://github.com/openzipkin/zipkin
>>>
>>> Thanks,
>>> Stephen
>>>
>>>
>>> On Mon, Aug 30, 2021 at 5:41 PM Nick Dimiduk 
>>> wrote:
>>> >
>>> > +0
>>> >
>>> > Recent PR test failures are unfortunate, but look unrelated. It's
>>> > unfortunate, because there's enough change here that I think we'd all
>>> be a
>>> > lot more comfortable with better confidence that the changeset has not
>>> > introduced instabilities.
>>> >
>>> > I think we're pretty close to cutting a 2.5.0 from this branch. I've
>>> not
>>> > seen anyone report their success at integrating these changes into a
>>> > telemetry system and showing end-to-end tracing of a service call. As I
>>> > recall from the last 2.5 discuss thread, we wanted this to be the
>>> banner
>>> > feature for the minor release. Since we don't have this end-to-end
>>> > confirmation, do we have someone who's volunteered to demonstrate that
>>> > final proof-of-integration in a timely manner? Perhaps that someone is
>>> keen
>>> > to be release manager for 2.5.
>>> >
>>> > I didn't see a discuss thread talking about what open items might be
>>> left
>>> > before merge, so I raise my question here. Pardon me if these comments
>>> > should have landed elsewhere.
>>> >
>>> > Thanks,
>>> > Nick
>>> >
>>> > On Sun, Aug 29, 2021 at 8:22 PM Tak Lon (Stephen) Wu <
>>> tak...@apache.org>
>>> > wrote:
>>> >
>>> > > Hi everyone,
>>> > >
>>> > > I'm writing this request and propose a merge of HBASE-25853
>>> > >  "Backport
>>> HBASE-22120
>>> > > Replace HTrace with OpenTelemetry" to branch-2. The goal is to remove
>>> > > HTrace and uses OpenTelemetry also in branch-2.5.0+.
>>> > >
>>> > > Highlights
>>> > > * These changes only support async clients and calls
>>> > > * We will have a separate thread for support sync client in branch-2
>>> , see
>>> > > HBASE-26141 
>>> > >
>>> > > PRs
>>> > > * The merge PR with 18 reviewed commits
>>> > > https://github.com/apache/hbase/pull/3637
>>> > > * see all those reviewed commits at feature branch HBASE-25853
>>

Re: [VOTE] Merge feature branch HBASE-25853 into branch-2

2021-09-01 Thread Tak Lon (Stephen) Wu 
FYI, I will be out of my desk this week, let's see if I can give a try next
week on https://issues.apache.org/jira/browse/HBASE-26241

On Wed, Sep 1, 2021 at 3:24 PM Tak Lon (Stephen) Wu 
wrote:

> Thanks Nick!
>
> for the vote to merge HBASE-25853 into branch-2, with 3 +1s and 1 +0 from
> committers, the vote passed and I'm going to merge the feature into
> branch-2.
>
> -Stephen
>
> On Wed, Sep 1, 2021 at 11:46 AM Nick Dimiduk  wrote:
>
>> This is great information. I would feel better about it if we had
>> confirmation that everything is working as expected on branch-2. The reason
>> for this is that master and branch-2 have drifted quite a bit, and the
>> instrumentation points are quite sensitive to internal details of the code.
>> Thus, I don’t assume that behavior demonstrated on master is necessarily
>> applicable to branch-2.
>>
>> Since no one seems to have demonstrated on the feature branch what I’m
>> hoping for, let me spend some time with the feature branch and/or branch-2
>> after merge. I think it’s important that we know this works well, as it is
>> a headline feature of 2.5. (If anyone else is really keen to do this
>> demonstration, please let us know!)
>>
>> Related: what’s the status is tracing into HDFS and beyond? Should I
>> expect to see spans all the way down to HDFS disk access? What about kernel
>> system calls?
>>
>> Thanks,
>> Nick
>>
>> On Mon, Aug 30, 2021 at 20:30 张铎(Duo Zhang) 
>> wrote:
>>
>>> Before I left Xiaomi, I’ve already set up a testing cluster and
>>> collected the tracing data to Apache Skywalking and show it.
>>>
>>> There are some screen shots, I found them in the private slack channel
>>> between Stack and me.
>>>
>>>
>>>
>>> Tak Lon (Stephen) Wu 于2021年8月31日 周二10:09写道：
>>>
 Hey Nick, IMO you're good, and we should move your concern to another
 discussion topic.

 Let me reply here first and see if the created JIRA
 https://issues.apache.org/jira/browse/HBASE-26241 (could be a blocker
 for hbase-2.5.0 release) meets your exception.

 If I read your message correctly, even if we have the instruction on
 how to enable opentelemetery for writing trace in HBase's log [1]
 (should have tested in HBASE-25658 [2]) as well as we have the
 interface for other telemetry sinks [3], you would like to see the
 following before releasing 2.5.0 if we're merging HBASE-25853 into
 branch-2?

 a. Someone (either the RM or me) can spend a bit on testing end-2-end
 with a graph as result in the JIRA (or hbase book html) that this
 tracing can be rendered at the sink of an external telemetry system
 e.g. Jaeger [4] or zipkin [5].
 b. For a `service call`, e.g. via a put/get call, how does the trace
 message look like in logs or via from the UI?

 [1] https://hbase.apache.org/book.html#tracing
 [2] https://issues.apache.org/jira/browse/HBASE-25658 has the basic
 benchmark
 [3]
 https://danw1ld.medium.com/observability-for-front-end-web-clients-with-opentelemetry-and-jaeger-in-5-minutes-343f719fbf5a
 [4] Jaeger, https://github.com/jaegertracing/jaeger
 [5] zipkin, https://github.com/openzipkin/zipkin

 Thanks,
 Stephen


 On Mon, Aug 30, 2021 at 5:41 PM Nick Dimiduk 
 wrote:
 >
 > +0
 >
 > Recent PR test failures are unfortunate, but look unrelated. It's
 > unfortunate, because there's enough change here that I think we'd all
 be a
 > lot more comfortable with better confidence that the changeset has not
 > introduced instabilities.
 >
 > I think we're pretty close to cutting a 2.5.0 from this branch. I've
 not
 > seen anyone report their success at integrating these changes into a
 > telemetry system and showing end-to-end tracing of a service call. As
 I
 > recall from the last 2.5 discuss thread, we wanted this to be the
 banner
 > feature for the minor release. Since we don't have this end-to-end
 > confirmation, do we have someone who's volunteered to demonstrate that
 > final proof-of-integration in a timely manner? Perhaps that someone
 is keen
 > to be release manager for 2.5.
 >
 > I didn't see a discuss thread talking about what open items might be
 left
 > before merge, so I raise my question here. Pardon me if these comments
 > should have landed elsewhere.
 >
 > Thanks,
 > Nick
 >
 > On Sun, Aug 29, 2021 at 8:22 PM Tak Lon (Stephen) Wu <
 tak...@apache.org>
 > wrote:
 >
 > > Hi everyone,
 > >
 > > I'm writing this request and propose a merge of HBASE-25853
 > >  "Backport
 HBASE-22120
 > > Replace HTrace with OpenTelemetry" to branch-2. The goal is to
 remove
 > > HTrace and uses OpenTelemetry also in branch-2.5.0+.
 > >
 > > Highlights
 > > * These changes only support async clients and calls
 > > * W

[jira] [Resolved] (HBASE-25853) Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2

2021-09-01 Thread Tak-Lon (Stephen) Wu (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Tak-Lon (Stephen) Wu resolved HBASE-25853.
--
Resolution: Fixed

https://github.com/apache/hbase/pull/3637 merged into branch-2

> Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
> 
>
> Key: HBASE-25853
> URL: https://issues.apache.org/jira/browse/HBASE-25853
> Project: HBase
>  Issue Type: Task
>  Components: tracing
>Affects Versions: 2.5.0
>Reporter: Duo Zhang
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> list of commits from upstream/master need to be backported 
> {code}
> HBASE-25373 
> https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef
>  
> HBASE-25401 
> https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04
>  
> HBASE-25424 
> https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d
>  
> HBASE-23898 
> https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59
>  
> HBASE-25454 
> https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93
> HBASE-25481 
> https://github.com/apache/hbase/commit/ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
> HBASE-25455 
> https://github.com/apache/hbase/commit/03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
> HBASE-25484 
> https://github.com/apache/hbase/commit/2be2c63f0d3917a243b74af9754cbfc805b858d1
> HBASE-25535 
> https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3
> HBASE-25591 
> https://github.com/apache/hbase/commit/f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
> HBASE-25617 
> https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294
> HBASE-25616 
> https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99
> HBASE-25723 
> https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea
> HBASE-25732 
> https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071
> HBASE-25733 
> https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673
> HBASE-23762 
> https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238
> HBASE-25778 
> https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a
> {code}



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Merge feature branch HBASE-25853 into branch-2

2021-09-01 Thread Tak Lon (Stephen) Wu 
Thanks Nick!

for the vote to merge HBASE-25853 into branch-2, with 3 +1s and 1 +0 from
committers, the vote passed and I'm going to merge the feature into
branch-2.

-Stephen

On Wed, Sep 1, 2021 at 11:46 AM Nick Dimiduk  wrote:

> This is great information. I would feel better about it if we had
> confirmation that everything is working as expected on branch-2. The reason
> for this is that master and branch-2 have drifted quite a bit, and the
> instrumentation points are quite sensitive to internal details of the code.
> Thus, I don’t assume that behavior demonstrated on master is necessarily
> applicable to branch-2.
>
> Since no one seems to have demonstrated on the feature branch what I’m
> hoping for, let me spend some time with the feature branch and/or branch-2
> after merge. I think it’s important that we know this works well, as it is
> a headline feature of 2.5. (If anyone else is really keen to do this
> demonstration, please let us know!)
>
> Related: what’s the status is tracing into HDFS and beyond? Should I
> expect to see spans all the way down to HDFS disk access? What about kernel
> system calls?
>
> Thanks,
> Nick
>
> On Mon, Aug 30, 2021 at 20:30 张铎(Duo Zhang)  wrote:
>
>> Before I left Xiaomi, I’ve already set up a testing cluster and collected
>> the tracing data to Apache Skywalking and show it.
>>
>> There are some screen shots, I found them in the private slack channel
>> between Stack and me.
>>
>>
>>
>> Tak Lon (Stephen) Wu 于2021年8月31日 周二10:09写道：
>>
>>> Hey Nick, IMO you're good, and we should move your concern to another
>>> discussion topic.
>>>
>>> Let me reply here first and see if the created JIRA
>>> https://issues.apache.org/jira/browse/HBASE-26241 (could be a blocker
>>> for hbase-2.5.0 release) meets your exception.
>>>
>>> If I read your message correctly, even if we have the instruction on
>>> how to enable opentelemetery for writing trace in HBase's log [1]
>>> (should have tested in HBASE-25658 [2]) as well as we have the
>>> interface for other telemetry sinks [3], you would like to see the
>>> following before releasing 2.5.0 if we're merging HBASE-25853 into
>>> branch-2?
>>>
>>> a. Someone (either the RM or me) can spend a bit on testing end-2-end
>>> with a graph as result in the JIRA (or hbase book html) that this
>>> tracing can be rendered at the sink of an external telemetry system
>>> e.g. Jaeger [4] or zipkin [5].
>>> b. For a `service call`, e.g. via a put/get call, how does the trace
>>> message look like in logs or via from the UI?
>>>
>>> [1] https://hbase.apache.org/book.html#tracing
>>> [2] https://issues.apache.org/jira/browse/HBASE-25658 has the basic
>>> benchmark
>>> [3]
>>> https://danw1ld.medium.com/observability-for-front-end-web-clients-with-opentelemetry-and-jaeger-in-5-minutes-343f719fbf5a
>>> [4] Jaeger, https://github.com/jaegertracing/jaeger
>>> [5] zipkin, https://github.com/openzipkin/zipkin
>>>
>>> Thanks,
>>> Stephen
>>>
>>>
>>> On Mon, Aug 30, 2021 at 5:41 PM Nick Dimiduk 
>>> wrote:
>>> >
>>> > +0
>>> >
>>> > Recent PR test failures are unfortunate, but look unrelated. It's
>>> > unfortunate, because there's enough change here that I think we'd all
>>> be a
>>> > lot more comfortable with better confidence that the changeset has not
>>> > introduced instabilities.
>>> >
>>> > I think we're pretty close to cutting a 2.5.0 from this branch. I've
>>> not
>>> > seen anyone report their success at integrating these changes into a
>>> > telemetry system and showing end-to-end tracing of a service call. As I
>>> > recall from the last 2.5 discuss thread, we wanted this to be the
>>> banner
>>> > feature for the minor release. Since we don't have this end-to-end
>>> > confirmation, do we have someone who's volunteered to demonstrate that
>>> > final proof-of-integration in a timely manner? Perhaps that someone is
>>> keen
>>> > to be release manager for 2.5.
>>> >
>>> > I didn't see a discuss thread talking about what open items might be
>>> left
>>> > before merge, so I raise my question here. Pardon me if these comments
>>> > should have landed elsewhere.
>>> >
>>> > Thanks,
>>> > Nick
>>> >
>>> > On Sun, Aug 29, 2021 at 8:22 PM Tak Lon (Stephen) Wu <
>>> tak...@apache.org>
>>> > wrote:
>>> >
>>> > > Hi everyone,
>>> > >
>>> > > I'm writing this request and propose a merge of HBASE-25853
>>> > >  "Backport
>>> HBASE-22120
>>> > > Replace HTrace with OpenTelemetry" to branch-2. The goal is to remove
>>> > > HTrace and uses OpenTelemetry also in branch-2.5.0+.
>>> > >
>>> > > Highlights
>>> > > * These changes only support async clients and calls
>>> > > * We will have a separate thread for support sync client in branch-2
>>> , see
>>> > > HBASE-26141 
>>> > >
>>> > > PRs
>>> > > * The merge PR with 18 reviewed commits
>>> > > https://github.com/apache/hbase/pull/3637
>>> > > * see all those reviewed commits at feature branch HBASE-25853

Re: [VOTE] Merge feature branch HBASE-25853 into branch-2

2021-09-01 Thread Nick Dimiduk 
This is great information. I would feel better about it if we had
confirmation that everything is working as expected on branch-2. The reason
for this is that master and branch-2 have drifted quite a bit, and the
instrumentation points are quite sensitive to internal details of the code.
Thus, I don’t assume that behavior demonstrated on master is necessarily
applicable to branch-2.

Since no one seems to have demonstrated on the feature branch what I’m
hoping for, let me spend some time with the feature branch and/or branch-2
after merge. I think it’s important that we know this works well, as it is
a headline feature of 2.5. (If anyone else is really keen to do this
demonstration, please let us know!)

Related: what’s the status is tracing into HDFS and beyond? Should I expect
to see spans all the way down to HDFS disk access? What about kernel system
calls?

Thanks,
Nick

On Mon, Aug 30, 2021 at 20:30 张铎(Duo Zhang)  wrote:

> Before I left Xiaomi, I’ve already set up a testing cluster and collected
> the tracing data to Apache Skywalking and show it.
>
> There are some screen shots, I found them in the private slack channel
> between Stack and me.
>
>
>
> Tak Lon (Stephen) Wu 于2021年8月31日 周二10:09写道：
>
>> Hey Nick, IMO you're good, and we should move your concern to another
>> discussion topic.
>>
>> Let me reply here first and see if the created JIRA
>> https://issues.apache.org/jira/browse/HBASE-26241 (could be a blocker
>> for hbase-2.5.0 release) meets your exception.
>>
>> If I read your message correctly, even if we have the instruction on
>> how to enable opentelemetery for writing trace in HBase's log [1]
>> (should have tested in HBASE-25658 [2]) as well as we have the
>> interface for other telemetry sinks [3], you would like to see the
>> following before releasing 2.5.0 if we're merging HBASE-25853 into
>> branch-2?
>>
>> a. Someone (either the RM or me) can spend a bit on testing end-2-end
>> with a graph as result in the JIRA (or hbase book html) that this
>> tracing can be rendered at the sink of an external telemetry system
>> e.g. Jaeger [4] or zipkin [5].
>> b. For a `service call`, e.g. via a put/get call, how does the trace
>> message look like in logs or via from the UI?
>>
>> [1] https://hbase.apache.org/book.html#tracing
>> [2] https://issues.apache.org/jira/browse/HBASE-25658 has the basic
>> benchmark
>> [3]
>> https://danw1ld.medium.com/observability-for-front-end-web-clients-with-opentelemetry-and-jaeger-in-5-minutes-343f719fbf5a
>> [4] Jaeger, https://github.com/jaegertracing/jaeger
>> [5] zipkin, https://github.com/openzipkin/zipkin
>>
>> Thanks,
>> Stephen
>>
>>
>> On Mon, Aug 30, 2021 at 5:41 PM Nick Dimiduk  wrote:
>> >
>> > +0
>> >
>> > Recent PR test failures are unfortunate, but look unrelated. It's
>> > unfortunate, because there's enough change here that I think we'd all
>> be a
>> > lot more comfortable with better confidence that the changeset has not
>> > introduced instabilities.
>> >
>> > I think we're pretty close to cutting a 2.5.0 from this branch. I've not
>> > seen anyone report their success at integrating these changes into a
>> > telemetry system and showing end-to-end tracing of a service call. As I
>> > recall from the last 2.5 discuss thread, we wanted this to be the banner
>> > feature for the minor release. Since we don't have this end-to-end
>> > confirmation, do we have someone who's volunteered to demonstrate that
>> > final proof-of-integration in a timely manner? Perhaps that someone is
>> keen
>> > to be release manager for 2.5.
>> >
>> > I didn't see a discuss thread talking about what open items might be
>> left
>> > before merge, so I raise my question here. Pardon me if these comments
>> > should have landed elsewhere.
>> >
>> > Thanks,
>> > Nick
>> >
>> > On Sun, Aug 29, 2021 at 8:22 PM Tak Lon (Stephen) Wu > >
>> > wrote:
>> >
>> > > Hi everyone,
>> > >
>> > > I'm writing this request and propose a merge of HBASE-25853
>> > >  "Backport
>> HBASE-22120
>> > > Replace HTrace with OpenTelemetry" to branch-2. The goal is to remove
>> > > HTrace and uses OpenTelemetry also in branch-2.5.0+.
>> > >
>> > > Highlights
>> > > * These changes only support async clients and calls
>> > > * We will have a separate thread for support sync client in branch-2
>> , see
>> > > HBASE-26141 
>> > >
>> > > PRs
>> > > * The merge PR with 18 reviewed commits
>> > > https://github.com/apache/hbase/pull/3637
>> > > * see all those reviewed commits at feature branch HBASE-25853
>> > > 
>> > >
>> > > Please vote:
>> > > [+1]  Agree
>> > > [+/-0]Neutral
>> > > [-1]   Disagree (please include actionable feedback)
>> > >
>> > > Thanks,
>> > > Stephen
>> > >
>>
>

[jira] [Created] (HBASE-26248) Should find a suitable way to let users specify the store file tracker implementation

2021-09-01 Thread Duo Zhang (Jira) 
Duo Zhang created HBASE-26248:
-

 Summary: Should find a suitable way to let users specify the store 
file tracker implementation
 Key: HBASE-26248
 URL: https://issues.apache.org/jira/browse/HBASE-26248
 Project: HBase
  Issue Type: Sub-task
  Components: HFile
Reporter: Duo Zhang


Now all the implementations are marked as IA.Private, so there is no safe way 
for users to specify them.

But for simplify, maybe we should not expose the full class name of the 
implementation classes to users. Just follow the way in WALFactory, to 
introduce an alias name for the implemention, for example

DEFAULT, FILE, REGION, etc.

Thoughts?



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26247) TestWALRecordReader#testWALRecordReaderActiveArchiveTolerance doesn't read archived WAL file.

2021-09-01 Thread Rushabh Shah (Jira) 
Rushabh Shah created HBASE-26247:


 Summary: 
TestWALRecordReader#testWALRecordReaderActiveArchiveTolerance doesn't read 
archived WAL file. 
 Key: HBASE-26247
 URL: https://issues.apache.org/jira/browse/HBASE-26247
 Project: HBase
  Issue Type: Bug
Reporter: Rushabh Shah


TestWALRecordReader#testWALRecordReaderActiveArchiveTolerance is testing the 
following scenario.
1. Create a new WAL file
2. Write 2 KVs to WAL file.
3. Close the WAL file.
4. Instantiate WALInputFormat#WALKeyRecordReader with the WAL created in step 1.
5. Read the first KV.
6. Archive the WAL file in oldWALs directory via rename operation.
7. Read the second KV. This will test that WALKeyRecordReader will encounter 
FNFE since the WAL file is not longer present in the original location and it 
will handle the FNFE by opening the WAL file from archived location.

In step#7, the test is expecting that it will encounter FNFE and it will open a 
new reader but in reality, it is not encountering FNFE. I think the reason is, 
during rename operation, HDFS just changes the internal metadata for the file 
name. The InodeID, hdfs blocks and block locations remains the same. While 
reading the first KV, DFSInputStream caches all the HDFS blocks and location 
data so it doesn't have to go to namenode to re-resolve the file name.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Resolved] (HBASE-26205) TableMRUtil#initCredentialsForCluster should use specified conf for UserProvider

2021-09-01 Thread Duo Zhang (Jira) 


 [ 
https://issues.apache.org/jira/browse/HBASE-26205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Duo Zhang resolved HBASE-26205.
---
Hadoop Flags: Reviewed
  Resolution: Fixed

Pushed to branch-2.3+.

Thanks [~lineyshinya] for contributing and [~shahrs87] for reviewing.

> TableMRUtil#initCredentialsForCluster should use specified conf for 
> UserProvider
> 
>
> Key: HBASE-26205
> URL: https://issues.apache.org/jira/browse/HBASE-26205
> Project: HBase
>  Issue Type: Bug
>  Components: mapreduce
>Reporter: Shinya Yoshida
>Assignee: Shinya Yoshida
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-2, 2.4.6, 2.3.7
>
>
> {code:java}
>   /**
>* Obtain an authentication token, for the specified cluster, on behalf of 
> the current user
>* and add it to the credentials for the given map reduce job.
>*
>* @param job The job that requires the permission.
>* @param conf The configuration to use in connecting to the peer cluster
>* @throws IOException When the authentication token cannot be obtained.
>*/
>   public static void initCredentialsForCluster(Job job, Configuration conf)
>   throws IOException {
> UserProvider userProvider = 
> UserProvider.instantiate(job.getConfiguration());
> if (userProvider.isHBaseSecurityEnabled()) {
>   try {
> Connection peerConn = ConnectionFactory.createConnection(conf);
> try {
>   TokenUtil.addTokenForJob(peerConn, userProvider.getCurrent(), job);
> } finally {
>   peerConn.close();
> }
>   } catch (InterruptedException e) {
> LOG.info("Interrupted obtaining user authentication token");
> Thread.interrupted();
>   }
> }
>   }
> {code}
> TableMapReduceUtil#initCredentialsForCluster uses job's configuration instead 
> of conf in argument for UserProvider.
> This causes that token is not obtained for secure cluster in case of job's 
> configuration contains hbase.security.authentication=simple.
> Because userProvider.isHBaseSecurityEnabled() checks that 
> hbase.security.authentication is kerberos.
> If conf is configured for secure cluster, hbase.security.authentication is 
> kerberos and it's natural to use conf for UserProvider instead of job's 
> configuration in this method.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-26246) Persist the store engine configuration to TableDescriptor when creating table

2021-09-01 Thread Duo Zhang (Jira) 
Duo Zhang created HBASE-26246:
-

 Summary: Persist the store engine configuration to TableDescriptor 
when creating table
 Key: HBASE-26246
 URL: https://issues.apache.org/jira/browse/HBASE-26246
 Project: HBase
  Issue Type: Sub-task
  Components: HFile
Reporter: Duo Zhang


As discussed in this section in the design doc:

https://docs.google.com/document/d/16Nr1Fn3VaXuz1g1FTiME-bnGR3qVK5B-raXshOkDLcY/edit#heading=h.78r2mdeyquug

If we use different SFT implementation at master side and region server side, 
it is likely to cause data loss, which is a very serious misconfiguration 
problem.

A possible solution is to make sure that master and region server always load 
the configurations about StoreEngine from the same place. To archive this, a 
possible way is to always set the StoreEngine configurations to the 
TableDescriptor, even if user does not explicitly set it when creating a table.

And also, when upgrading, we should check whether the existing tables have 
StoreEngine configurations, if not, we need to set them.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)