[jira] [Resolved] (HBASE-28161) Release hbase-connectors 1.0.1
[ https://issues.apache.org/jira/browse/HBASE-28161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28161. -- Resolution: Fixed > Release hbase-connectors 1.0.1 > -- > > Key: HBASE-28161 > URL: https://issues.apache.org/jira/browse/HBASE-28161 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28162) Complete release hbase-connectors 1.0.1
[ https://issues.apache.org/jira/browse/HBASE-28162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28162. -- Release Note: added release note at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753=12345436 Resolution: Fixed > Complete release hbase-connectors 1.0.1 > --- > > Key: HBASE-28162 > URL: https://issues.apache.org/jira/browse/HBASE-28162 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > # Release the artifacts on repository.apache.org > # Move the binaries from dist-dev to dist-release > # Add xml to download page > # Push tag 1.0.1RC as tag rel/1.0.1 > # Release 1.0.1 on JIRA > https://issues.apache.org/jira/projects/HBASE/versions/12345436 > # Add release data on [https://reporter.apache.org/addrelease.html?hbase] > # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28164) Add hbase-connectos 1.0.1 to download page
[ https://issues.apache.org/jira/browse/HBASE-28164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28164. -- Resolution: Fixed > Add hbase-connectos 1.0.1 to download page > -- > > Key: HBASE-28164 > URL: https://issues.apache.org/jira/browse/HBASE-28164 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 4.0.0-alpha-1 > > > need a PR to add hbase-connectors rel/1.0.1 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28163) Put up hbase-connectors 1.0.1RC0
[ https://issues.apache.org/jira/browse/HBASE-28163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28163. -- Hadoop Flags: Reviewed Resolution: Fixed > Put up hbase-connectors 1.0.1RC0 > - > > Key: HBASE-28163 > URL: https://issues.apache.org/jira/browse/HBASE-28163 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28165) Fix hbase-connectors for release scripts
Tak-Lon (Stephen) Wu created HBASE-28165: Summary: Fix hbase-connectors for release scripts Key: HBASE-28165 URL: https://issues.apache.org/jira/browse/HBASE-28165 Project: HBase Issue Type: Bug Reporter: Tak-Lon (Stephen) Wu During the release for hbase-connectors 1.0.1 HBASE-28161, I found several issues 1. CHANGELOG.md was being used but our release script is using CHANGES.md to create new changes. 2. the do-release.sh script isn't working because of due 1/, I hacked it to make it work, but this would bring a backward compatible issues. Proposing to fix it after the releases, because no matter what the archive does not have the CHANGES.md for hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28163) CLONE - Put up 2.4.17RC0
Tak-Lon (Stephen) Wu created HBASE-28163: Summary: CLONE - Put up 2.4.17RC0 Key: HBASE-28163 URL: https://issues.apache.org/jira/browse/HBASE-28163 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28161) Release hbase-connectors 1.0.1
Tak-Lon (Stephen) Wu created HBASE-28161: Summary: Release hbase-connectors 1.0.1 Key: HBASE-28161 URL: https://issues.apache.org/jira/browse/HBASE-28161 Project: HBase Issue Type: Umbrella Components: community Reporter: Duo Zhang Assignee: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28164) CLONE - Add 2.4.17 to download page
Tak-Lon (Stephen) Wu created HBASE-28164: Summary: CLONE - Add 2.4.17 to download page Key: HBASE-28164 URL: https://issues.apache.org/jira/browse/HBASE-28164 Project: HBase Issue Type: Sub-task Components: website Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu Fix For: 3.0.0-alpha-4 need a PR to add rel/2.4.17 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28162) CLONE - Complete Release 2.4.17
Tak-Lon (Stephen) Wu created HBASE-28162: Summary: CLONE - Complete Release 2.4.17 Key: HBASE-28162 URL: https://issues.apache.org/jira/browse/HBASE-28162 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu # Release the artifacts on repository.apache.org # Move the binaries from dist-dev to dist-release # Add xml to download page # Push tag 2.4.17RC0 as tag rel/2.4.17 # Release 2.4.17 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12352760 # Add release data on [https://reporter.apache.org/addrelease.html?hbase] # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27759) Release 2.4.17
[ https://issues.apache.org/jira/browse/HBASE-27759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27759. -- Resolution: Fixed > Release 2.4.17 > -- > > Key: HBASE-27759 > URL: https://issues.apache.org/jira/browse/HBASE-27759 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28055) Performance improvement for scan over several stores.
[ https://issues.apache.org/jira/browse/HBASE-28055?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28055. -- Resolution: Fixed pushed to master/4.0.0-alpha-1, branch-3, branch-2, branch-2.5, branch-2.4, resolving it. > Performance improvement for scan over several stores. > -- > > Key: HBASE-28055 > URL: https://issues.apache.org/jira/browse/HBASE-28055 > Project: HBase > Issue Type: Bug >Affects Versions: 3.0.0-alpha-4, 2.5.5 >Reporter: Sergey Soldatov >Assignee: Sergey Soldatov >Priority: Major > Fix For: 2.6.0, 2.4.18, 2.5.6, 3.0.0-beta-1, 4.0.0-alpha-1 > > > During the fix of HBASE-19863, an additional check for fake cells that > trigger reseek was added. It comes that this check produces unnecessary > reseeks because > matcher.compareKeyForNextColumn should be used only with indexed keys. Later > [~larsh] suggested doing a simple check for OLD_TIMESTAMP and it looks like a > better solution. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27839) Backport HBASE-27091 to branch-2
Tak-Lon (Stephen) Wu created HBASE-27839: Summary: Backport HBASE-27091 to branch-2 Key: HBASE-27839 URL: https://issues.apache.org/jira/browse/HBASE-27839 Project: HBase Issue Type: Task Components: master Affects Versions: 2.5.4, 2.4.17 Reporter: Tak-Lon (Stephen) Wu HBASE-27091 could be useful to branch-2 as well, creating this task for backporting from master -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27770) Complete Release 2.4.17
[ https://issues.apache.org/jira/browse/HBASE-27770?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27770. -- Resolution: Fixed Complete the download page update, as well as sending out the announcement email for releasing 2.4.17 > Complete Release 2.4.17 > --- > > Key: HBASE-27770 > URL: https://issues.apache.org/jira/browse/HBASE-27770 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > # Release the artifacts on repository.apache.org > # Move the binaries from dist-dev to dist-release > # Add xml to download page > # Push tag 2.4.17RC0 as tag rel/2.4.17 > # Release 2.4.17 on JIRA > https://issues.apache.org/jira/projects/HBASE/versions/12352760 > # Add release data on [https://reporter.apache.org/addrelease.html?hbase] > # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27772) Add 2.4.17 to download page
[ https://issues.apache.org/jira/browse/HBASE-27772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27772. -- Fix Version/s: 3.0.0-alpha-4 Resolution: Fixed code merged > Add 2.4.17 to download page > --- > > Key: HBASE-27772 > URL: https://issues.apache.org/jira/browse/HBASE-27772 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 3.0.0-alpha-4 > > > need a PR to add rel/2.4.17 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27771) Put up 2.4.17RC0
[ https://issues.apache.org/jira/browse/HBASE-27771?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27771. -- Resolution: Fixed completed the RC vote > Put up 2.4.17RC0 > - > > Key: HBASE-27771 > URL: https://issues.apache.org/jira/browse/HBASE-27771 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27772) Add 2.4.17 to download page
Tak-Lon (Stephen) Wu created HBASE-27772: Summary: Add 2.4.17 to download page Key: HBASE-27772 URL: https://issues.apache.org/jira/browse/HBASE-27772 Project: HBase Issue Type: Sub-task Components: website Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu need a PR to add rel/2.4.17 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27771) Put up 2.4.17RC0
Tak-Lon (Stephen) Wu created HBASE-27771: Summary: Put up 2.4.17RC0 Key: HBASE-27771 URL: https://issues.apache.org/jira/browse/HBASE-27771 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27770) Complete Release 2.4.17
Tak-Lon (Stephen) Wu created HBASE-27770: Summary: Complete Release 2.4.17 Key: HBASE-27770 URL: https://issues.apache.org/jira/browse/HBASE-27770 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu # Release the artifacts on repository.apache.org # Move the binaries from dist-dev to dist-release # Add xml to download page # Push tag 2.4.17RC0 as tag rel/2.4.17 # Release 2.4.17 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12352760 # Add release data on [https://reporter.apache.org/addrelease.html?hbase] # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27612) Push 2.4.16 release out
[ https://issues.apache.org/jira/browse/HBASE-27612?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27612. -- Resolution: Fixed > Push 2.4.16 release out > --- > > Key: HBASE-27612 > URL: https://issues.apache.org/jira/browse/HBASE-27612 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > # Release the artifacts on repository.apache.org > # Move the binaries from dist-dev to dist-release > # Add to download page, which is tracked by HBASE-27610 as it requires code > change > # Push tag 2.4.16RC1 as tag rel/2.4.16 > # Release 2.4.16 on jira > # Add release data on https://reporter.apache.org/addrelease.html?hbase > # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27604) Release 2.4.16
[ https://issues.apache.org/jira/browse/HBASE-27604?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27604. -- Resolution: Fixed > Release 2.4.16 > -- > > Key: HBASE-27604 > URL: https://issues.apache.org/jira/browse/HBASE-27604 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27769) Use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for non-HDFS file system
Tak-Lon (Stephen) Wu created HBASE-27769: Summary: Use hasPathCapability to support recoverLease, setSafeMode, isFileClosed for non-HDFS file system Key: HBASE-27769 URL: https://issues.apache.org/jira/browse/HBASE-27769 Project: HBase Issue Type: Improvement Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27626) Suppress noisy logging in client.ConnectionImplementation
[
https://issues.apache.org/jira/browse/HBASE-27626?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27626.
--
Hadoop Flags: Reviewed
Resolution: Fixed
> Suppress noisy logging in client.ConnectionImplementation
> -
>
> Key: HBASE-27626
> URL: https://issues.apache.org/jira/browse/HBASE-27626
> Project: HBase
> Issue Type: Task
> Components: logging
>Affects Versions: 2.4.16, 2.5.3
>Reporter: Nihal Jain
>Assignee: Nihal Jain
>Priority: Minor
> Fix For: 2.6.0, 2.4.17, 2.5.4
>
>
> _client.ConnectionImplementation_ logs a lot at INFO level:
> {code:java}
> hbase:001:0> restore_snapshot 'tableSnapshot'
> 2023-02-09 05:15:35,538 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:35,538 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:36,599 INFO [main] client.HBaseAdmin: Taking
> restore-failsafe snapshot: hbase-failsafe-tableSnapshot-1675919736599
> 2023-02-09 05:15:36,608 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:36,608 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:36,806 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:36,806 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,026 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,026 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,334 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,334 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,341 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,342 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,413 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,413 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,534 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,535 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:37,742 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:37,742 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:38,055 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:38,056 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:38,561 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:38,561 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-02-09 05:15:38,568 INFO [main] client.HBaseAdmin: Operation: MODIFY,
> Table Name: nj:testImport, procId: 392 completed
> 2023-02-09 05:15:38,568 INFO [main] client.HBaseAdmin: Deleting
> restore-failsafe snapshot: hbase-failsafe-tableSnapshot-1675919736599
> 2023-02-09 05:15:38,570 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-02-09 05:15:38,570 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> Took 3.1595 seconds
> {code}
> We should log these lines in TRACE level.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27603) Optimize INFO message after HBASE-27498
[
https://issues.apache.org/jira/browse/HBASE-27603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27603.
--
Resolution: Duplicate
> Optimize INFO message after HBASE-27498
> ---
>
> Key: HBASE-27603
> URL: https://issues.apache.org/jira/browse/HBASE-27603
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 2.4.16, 2.5.3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>
> During the RC vote of 2.4.16 and 2.5.3, I found the below message should be
> removed or hidden from the connection operation.
> {code}
> 2023-01-31T15:19:03,890 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-01-31T15:19:03,891 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-01-31T15:19:03,997 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-01-31T15:19:03,997 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-01-31T15:19:04,201 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-01-31T15:19:04,202 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> 2023-01-31T15:19:04,508 INFO [main] client.ConnectionImplementation: Getting
> master connection state from TTL Cache
> 2023-01-31T15:19:04,508 INFO [main] client.ConnectionImplementation: Getting
> master state using rpc call
> {code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27609) Release 2.5.3
[ https://issues.apache.org/jira/browse/HBASE-27609?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27609. -- Resolution: Fixed Thanks a lot for Andrew and Duo for guiding and helping, 2.5.3 has been released > Release 2.5.3 > - > > Key: HBASE-27609 > URL: https://issues.apache.org/jira/browse/HBASE-27609 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27613) Complete Release 2.5.3
[ https://issues.apache.org/jira/browse/HBASE-27613?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27613. -- Resolution: Fixed it's updated, thanks again Duo, I'm resolving it. https://hbase.apache.org/downloads.html > Complete Release 2.5.3 > -- > > Key: HBASE-27613 > URL: https://issues.apache.org/jira/browse/HBASE-27613 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > # Release the artifacts on repository.apache.org > # Move the binaries from dist-dev to dist-release > # HBASE-27611 Add to download page > # Push tag 2.5.3RC1 as tag rel/2.5.3 > # Release 2.5.3 on JIRA > [https://issues.apache.org/jira/projects/HBASE/versions/12352572] > # Add release data on [https://reporter.apache.org/addrelease.html?hbase] > # Send announcement email > (learnt from HBASE-27612) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27611) update download.xml for release 2.5.3
[ https://issues.apache.org/jira/browse/HBASE-27611?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27611. -- Fix Version/s: 3.0.0-alpha-4 Hadoop Flags: Reviewed Resolution: Fixed > update download.xml for release 2.5.3 > - > > Key: HBASE-27611 > URL: https://issues.apache.org/jira/browse/HBASE-27611 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Minor > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27613) Complete Release 2.5.3
Tak-Lon (Stephen) Wu created HBASE-27613: Summary: Complete Release 2.5.3 Key: HBASE-27613 URL: https://issues.apache.org/jira/browse/HBASE-27613 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu # Release the artifacts on repository.apache.org # Move the binaries from dist-dev to dist-release # HBASE-27611 Add to download page # Push tag 2.5.3RC1 as tag rel/2.5.3 # Release 2.5.3 on JIRA [https://issues.apache.org/jira/projects/HBASE/versions/12352572] # Add release data on [https://reporter.apache.org/addrelease.html?hbase] # Send announcement email (learn from HBASE-27612) -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27611) update download.xml for release 2.5.3
Tak-Lon (Stephen) Wu created HBASE-27611: Summary: update download.xml for release 2.5.3 Key: HBASE-27611 URL: https://issues.apache.org/jira/browse/HBASE-27611 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27609) Release 2.5.3
Tak-Lon (Stephen) Wu created HBASE-27609: Summary: Release 2.5.3 Key: HBASE-27609 URL: https://issues.apache.org/jira/browse/HBASE-27609 Project: HBase Issue Type: Umbrella Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27603) Optimize INFO message after HBASE-27498
Tak-Lon (Stephen) Wu created HBASE-27603:
Summary: Optimize INFO message after HBASE-27498
Key: HBASE-27603
URL: https://issues.apache.org/jira/browse/HBASE-27603
Project: HBase
Issue Type: Improvement
Affects Versions: 2.4.16, 2.5.3
Reporter: Tak-Lon (Stephen) Wu
Assignee: Tak-Lon (Stephen) Wu
During the RC vote of 2.4.16 and 2.5.3, I found the below message should be
removed or hidden from the connection operation.
{code}
2023-01-31T15:19:03,890 INFO [main] client.ConnectionImplementation: Getting
master connection state from TTL Cache
2023-01-31T15:19:03,891 INFO [main] client.ConnectionImplementation: Getting
master state using rpc call
2023-01-31T15:19:03,997 INFO [main] client.ConnectionImplementation: Getting
master connection state from TTL Cache
2023-01-31T15:19:03,997 INFO [main] client.ConnectionImplementation: Getting
master state using rpc call
2023-01-31T15:19:04,201 INFO [main] client.ConnectionImplementation: Getting
master connection state from TTL Cache
2023-01-31T15:19:04,202 INFO [main] client.ConnectionImplementation: Getting
master state using rpc call
2023-01-31T15:19:04,508 INFO [main] client.ConnectionImplementation: Getting
master connection state from TTL Cache
2023-01-31T15:19:04,508 INFO [main] client.ConnectionImplementation: Getting
master state using rpc call
{code}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Created] (HBASE-27589) Rename TestConnectionImplementationCacheMasterState to fix javadoc failure
Tak-Lon (Stephen) Wu created HBASE-27589: Summary: Rename TestConnectionImplementationCacheMasterState to fix javadoc failure Key: HBASE-27589 URL: https://issues.apache.org/jira/browse/HBASE-27589 Project: HBase Issue Type: Bug Components: Client, documentation Affects Versions: 2.6.0, 2.4.16, 2.5.3 Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu discussed during release of 2.5.3RC0 [https://lists.apache.org/thread/b34lgz3yy8vkv4fbbxj4mvtjyjrp4m6p] I found a javadoc problem introduced by HBASE-27498 with the same test class shared the same package prefix such that the javadoc failed with a unclear java.lang.NullPointerException. - hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java - hbase-server/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java after renaming hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementation.java to hbase-it/src/test/java/org/apache/hadoop/hbase/client/TestConnectionImplementationCacheMasterState.java should fix this issue. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27539) Encapsulate and centralise access to ref count through StoreFileInfo
[
https://issues.apache.org/jira/browse/HBASE-27539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27539.
--
Fix Version/s: (was: 2.5.3)
Hadoop Flags: Reviewed
Resolution: Fixed
thanks everyone, I have reverted it on branch-2.5 as of
[#392bc972eb5943536ac5ea4b1e9b16b3f7c51d89|https://github.com/apache/hbase/commit/392bc972eb5943536ac5ea4b1e9b16b3f7c51d89]
> Encapsulate and centralise access to ref count through StoreFileInfo
>
>
> Key: HBASE-27539
> URL: https://issues.apache.org/jira/browse/HBASE-27539
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-3
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> Both {{StoreFileReader}} and {{StoreFileInfo}} have a {{refCount}}, and the
> {{refCount}} is currently used in three main ways:
> * When a new {{StoreFileScanner}} is created or close, it increases or
> decreases the {{StoreFileReader.refCount}}.
> * When {{CompactedHFilesDischarger}} checks a {{HStoreFile}} whether it
> could be deleted, it check the {{StoreFileInfo.refCount}}.
> * When {{HStore.getScanners}} gets {{HStoreFile}} from {{StoreFileManager}},
> it increases or decreases the {{StoreFileInfo.refCount}}.
> The problem here is {{StoreFileReader.refCount}} is copied from the
> {{StoreFileInfo.refCount}} and the inconsistent usage of the {{refCount}}
> making the code somewhat hard to understand and causing trace the resource
> race problems such as HBASE-27484 and HBASE-27519 somewhat difficult. I
> suggest we should unify these two {{refCount}} and just use
> {{StoreFileInfo.refCount}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Reopened] (HBASE-27539) Encapsulate and centralise access to ref count through StoreFileInfo
[
https://issues.apache.org/jira/browse/HBASE-27539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu reopened HBASE-27539:
--
hi [~comnetwork] , [~wchevreuil] , we're preparing for the 2.5.3 release and I
found a back-compatible issue that
[#4939|https://github.com/apache/hbase/pull/4939] is breaking the API
compatibility for the upcoming patch version, especially the public constructor
of StoreFileReader is being used by phoenix (scope as IA.LimitedPrivate), and
it cannot break the compatibility for patch release.
Do you guys have any concerns if I revert this change for 2.5.x ?
> Encapsulate and centralise access to ref count through StoreFileInfo
>
>
> Key: HBASE-27539
> URL: https://issues.apache.org/jira/browse/HBASE-27539
> Project: HBase
> Issue Type: Improvement
>Affects Versions: 3.0.0-alpha-3
>Reporter: chenglei
>Assignee: chenglei
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.3
>
>
> Both {{StoreFileReader}} and {{StoreFileInfo}} have a {{refCount}}, and the
> {{refCount}} is currently used in three main ways:
> * When a new {{StoreFileScanner}} is created or close, it increases or
> decreases the {{StoreFileReader.refCount}}.
> * When {{CompactedHFilesDischarger}} checks a {{HStoreFile}} whether it
> could be deleted, it check the {{StoreFileInfo.refCount}}.
> * When {{HStore.getScanners}} gets {{HStoreFile}} from {{StoreFileManager}},
> it increases or decreases the {{StoreFileInfo.refCount}}.
> The problem here is {{StoreFileReader.refCount}} is copied from the
> {{StoreFileInfo.refCount}} and the inconsistent usage of the {{refCount}}
> making the code somewhat hard to understand and causing trace the resource
> race problems such as HBASE-27484 and HBASE-27519 somewhat difficult. I
> suggest we should unify these two {{refCount}} and just use
> {{StoreFileInfo.refCount}}
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27527) Port HBASE-27498 to branch-2
[ https://issues.apache.org/jira/browse/HBASE-27527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27527. -- Fix Version/s: 2.6.0 Hadoop Flags: Reviewed Release Note: see HBASE-27598 Resolution: Fixed [https://github.com/apache/hbase/commit/2e94f6fb504737ffdfa2567e6e2a9ea5aa07b8b8] merged . > Port HBASE-27498 to branch-2 > > > Key: HBASE-27527 > URL: https://issues.apache.org/jira/browse/HBASE-27527 > Project: HBase > Issue Type: Bug > Components: Client >Affects Versions: 2.0.0 >Reporter: Vaibhav Joshi >Priority: Major > Fix For: 2.6.0 > > > Port HBASE-27498 to "branch-2" > Please refer to PR [https://github.com/apache/hbase/pull/4919] > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27498) Observed lot of threads blocked in ConnectionImplementation.getKeepAliveMasterService
[
https://issues.apache.org/jira/browse/HBASE-27498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27498.
--
Fix Version/s: 2.4.16
2.5.3
Resolution: Fixed
> Observed lot of threads blocked in
> ConnectionImplementation.getKeepAliveMasterService
> -
>
> Key: HBASE-27498
> URL: https://issues.apache.org/jira/browse/HBASE-27498
> Project: HBase
> Issue Type: Bug
> Components: Client
>Affects Versions: 2.5.0
>Reporter: Vaibhav Joshi
>Priority: Major
> Fix For: 2.4.16, 2.5.3
>
> Attachments: Screenshot 2022-11-16 at 10.06.59 AM.png
>
>
> Recently We observed that lot of threads are blocked in method
> "ConnectionImplementation.getKeepAliveMasterService" during some
> initialization stages of rolling restart workflow.
> During rolling restart, we make RPC calls to Master using
> RpcRetryingCallerImpl, so as part of initialization we call
> "ConnectionImplementation.getKeepAliveMasterService" for each thread.
> Internally this method do RPC call within a synchronized block to check if
> master is running (mss.isMasterRunning).
> Lots of threads are in blocked state due following synchronized block
> synchronized (masterLock) {
> if (!isKeepAliveMasterConnectedAndRunning(this.masterServiceState))
> { MasterServiceStubMaker stubMaker = new MasterServiceStubMaker();
> this.masterServiceState.stub = stubMaker.makeStub(); }
> resetMasterServiceState(this.masterServiceState);
> }
> In Thread Dump Analyzer (2.4), we get warning that "A lot of threads are
> waiting for this monitor to become available again.
> This might indicate a congestion. You also should analyze other locks
> blocked by threads waiting for this monitor as there might be much more
> threads waiting for it.". Please check attached screenshot !Screenshot
> 2022-11-16 at 10.06.59 AM.png|width=1639,height=971!
>
> "pool-11-thread-158" #313 prio=5 os_prio=0 tid=0x55b88bcb8800 nid=0x404e
> waiting for monitor entry [0x7fa48aa86000]
> java.lang.Thread.State: BLOCKED (on object monitor)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getKeepAliveMasterService(ConnectionImplementation.java:1336)
> - waiting to lock <0x0005d30ecb68> (a java.lang.Object)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.getMaster(ConnectionImplementation.java:1327)
> at
> org.apache.hadoop.hbase.client.MasterCallable.prepare(MasterCallable.java:57)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:103)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3019)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3011)
> at org.apache.hadoop.hbase.client.HBaseAdmin.move(HBaseAdmin.java:1458)
> at
> org.apache.hadoop.hbase.util.MoveWithoutAck.call(MoveWithoutAck.java:58)
> at
> org.apache.hadoop.hbase.util.MoveWithoutAck.call(MoveWithoutAck.java:33)
> ---
>
> *Proposal:*
> We can optimize this flow as follows
> 1. Use double checked lock for
> "isKeepAliveMasterConnectedAndRunning(this.masterServiceState)" so that
> theads don't race for monitor, when master is running.
> 2. "isKeepAliveMasterConnectedAndRunning()" method should reuse the Globally
> cached state of isMasterRunning instead of doing expensive Call in for each
> thread.
> Check PR [https://github.com/apache/hbase/pull/4889] for more details.
> Note: The "master" branch uses "AsyncConnectionImpl" so apparently we don't
> have issues there.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27495) Improve HFileLinkCleaner to validate back reference links ahead the next traverse
[ https://issues.apache.org/jira/browse/HBASE-27495?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27495. -- Hadoop Flags: Reviewed Resolution: Fixed > Improve HFileLinkCleaner to validate back reference links ahead the next > traverse > -- > > Key: HBASE-27495 > URL: https://issues.apache.org/jira/browse/HBASE-27495 > Project: HBase > Issue Type: Improvement > Components: master >Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.5.2 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.6.0, 3.0.0-alpha-4, 2.5.2 > > > We found a a race in the CleanerChore related to back reference links. When > the HFileLinkCleaner runs for a file it can make 2 decisions depending on the > file types. > - HFiles, The cleaner for HFile deletion only checks if the .links-<> > directory is present with files. > - Back reference links, the cleaner checks if the forward link is still > available in the data directory. > The logic and order how the cleaner checks these 2 files matters. When the > back reference is checked first it can remove both the reference and the > HFile from the archive, however, when it first runs for the HFile then only > the back-reference is removed. In this case, the HFile is only deleted in the > next iteration of the CleanerChore, and it could be very slow if the list of > files are huge in case of using object store. > The goal of this task is to improve traverse of the archived HFile, reusing > the list of found back reference files, and immediately apply the checks for > the Back reference links. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27495) Improve HFileLinkCleaner to validate back reference links ahead the next traverse
Tak-Lon (Stephen) Wu created HBASE-27495: Summary: Improve HFileLinkCleaner to validate back reference links ahead the next traverse Key: HBASE-27495 URL: https://issues.apache.org/jira/browse/HBASE-27495 Project: HBase Issue Type: Improvement Affects Versions: 2.6.0, 3.0.0-alpha-4, 2.5.2 Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu We found a a race in the CleanerChore related to back reference links. When the HFileLinkCleaner runs for a file it can make 2 decisions depending on the file types. - Hfiles, The cleaner for HFile deletion only checks if the .links-<> directory is present with files. - Back reference links, the cleaner checks if the forward link is still available in the data directory. The logic and order how the cleaner checks these 2 files matters. When the back reference is checked first it can remove both the reference and the HFile from the archive, however, when it first runs for the HFile then only the back-reference is removed. In this case, the HFile is only deleted in the next iteration of the CleanerChore, and it could be very slow if the list of files are huge in case of using object store. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-24286) HMaster won't become healthy after after cloning or creating a new cluster pointing at the same file system
[
https://issues.apache.org/jira/browse/HBASE-24286?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-24286.
--
Resolution: Won't Fix
> HMaster won't become healthy after after cloning or creating a new cluster
> pointing at the same file system
> ---
>
> Key: HBASE-24286
> URL: https://issues.apache.org/jira/browse/HBASE-24286
> Project: HBase
> Issue Type: Bug
> Components: master, Region Assignment
>Affects Versions: 3.0.0-alpha-1, 2.2.3, 2.2.4, 2.2.5
>Reporter: Jack Ye
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
>
> h1. How to reproduce:
> # user starts an HBase cluster on top of a file system
> # user performs some operations and shuts down the cluster, all the data are
> still persisted in the file system
> # user creates a new HBase cluster using a different set of servers on top
> of the same file system with the same root directory
> # HMaster cannot initialize
> h1. Root cause:
> During HMaster initialization phase, the following happens:
> # HMaster waits for namespace table online
> # AssignmentManager gets all namespace table regions info
> # region servers of namespace table are already dead, online check fails
> # HMaster waits for namespace regions online, keep retrying for 1000 times
> which means forever
> Code waiting for namespace table to be online:
> https://github.com/apache/hbase/blob/rel/2.2.3/hbase-server/src/main/java/org/apache/hadoop/hbase/master/HMaster.java#L1102
> h1. Stack trace (running on S3):
> 2020-04-23 08:15:57,185 WARN [master/ip-10-12-13-14:16000:becomeActiveMaster]
> master.HMaster:
> hbase:namespace,,1587628169070.d34b65b91a52644ed3e77c5fbb065c2b. is NOT
> online; state=\{d34b65b91a52644ed3e77c5fbb065c2b state=OPEN,
> ts=1587629742129, server=ip-10-12-13-14.ec2.internal,16020,1587628031614};
> ServerCrashProcedures=false. Master startup cannot progress, in
> holding-pattern until region onlined.
> where ip-10-12-13-14.ec2.internal is the old region server hosting the region
> of hbase:namespace.
> h1. Discussion for the fix
> We see there is a fix for this at branch-3:
> https://issues.apache.org/jira/browse/HBASE-21154. Before we provide a patch,
> we would like to know from the community if we should backport this change to
> branch-2, or if we should just perform a fix with minimum code change.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Resolved] (HBASE-27295) Correct the comment of list_deadservers method in admin.rb
[ https://issues.apache.org/jira/browse/HBASE-27295?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27295. -- Hadoop Flags: Reviewed Resolution: Fixed > Correct the comment of list_deadservers method in admin.rb > -- > > Key: HBASE-27295 > URL: https://issues.apache.org/jira/browse/HBASE-27295 > Project: HBase > Issue Type: Improvement > Components: shell >Affects Versions: 3.0.0-alpha-4 >Reporter: LiangJun He >Assignee: LiangJun He >Priority: Minor > Fix For: 3.0.0-alpha-4 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-27297) Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2.5
Tak-Lon (Stephen) Wu created HBASE-27297: Summary: Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2.5 Key: HBASE-27297 URL: https://issues.apache.org/jira/browse/HBASE-27297 Project: HBase Issue Type: Improvement Components: conf, Coprocessors Affects Versions: 2.5.0, 2.5.1 Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu Backport HBASE-26810 dynamic configuration for system coprocessor to branch-2.5 , this have been done in master and branch-2, so let's see if we can have it in branch-2.5 before 2.5.0 or 2.5.1 release -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-27143) Add hbase-unsafe as a dependency for a MR job triggered by hbase shell
[
https://issues.apache.org/jira/browse/HBASE-27143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27143.
--
Fix Version/s: 2.5.0
3.0.0-alpha-3
2.4.13
Hadoop Flags: Reviewed
Resolution: Fixed
commits pushed to master, branch-2, branch-2.5, branch-2.4
> Add hbase-unsafe as a dependency for a MR job triggered by hbase shell
> --
>
> Key: HBASE-27143
> URL: https://issues.apache.org/jira/browse/HBASE-27143
> Project: HBase
> Issue Type: Bug
>Affects Versions: 2.5.0, 3.0.0-alpha-3, 2.4.13
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-3, 2.4.13
>
>
> After HBASE-26523, the use of hbase-thirdparty 4.1.0 introduced an new
> dependency of hbase-unsafe that wasn't added as part of the MR job triggered
> by hbase-shell. As such, class not found exception was being emitted when
> executing in distributed mode.
> {code}
> 22/06/21 04:57:15 INFO mapreduce.Job: Task Id :
> attempt_1655784404826_0011_m_00_0, Status : FAILED
> Error: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.unsafe.HBasePlatformDependent
> at
> java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
> at
> java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
> at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
> at org.apache.hadoop.hbase.util.Bytes.(Bytes.java:129)
> at
> org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:455)
> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.base/java.security.AccessController.doPrivileged(Native Method)
> at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> {code}
> This task will add it back as part of the job configuration handled by hbase
> internally in addHBaseDependencyJars
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27143) Add hbase-unsafe as a dependency for a MR job triggered by hbase shell
Tak-Lon (Stephen) Wu created HBASE-27143:
Summary: Add hbase-unsafe as a dependency for a MR job triggered
by hbase shell
Key: HBASE-27143
URL: https://issues.apache.org/jira/browse/HBASE-27143
Project: HBase
Issue Type: Bug
Affects Versions: 2.5.0, 3.0.0-alpha-3, 2.4.13
Reporter: Tak-Lon (Stephen) Wu
Assignee: Tak-Lon (Stephen) Wu
After HBASE-26523, the use of hbase-thirdparty 4.1.0 introduced an new
dependency of hbase-unsafe that wasn't added as part of the MR job triggered by
hbase-shell. As such, class not found exception was being emitted when
executing in distributed mode.
{code}
22/06/21 04:57:15 INFO mapreduce.Job: Task Id :
attempt_1655784404826_0011_m_00_0, Status : FAILED
Error: java.lang.ClassNotFoundException:
org.apache.hadoop.hbase.unsafe.HBasePlatformDependent
at
java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:583)
at
java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:521)
at org.apache.hadoop.hbase.util.Bytes.(Bytes.java:129)
at
org.apache.hadoop.hbase.test.IntegrationTestBigLinkedList$Generator$GeneratorMapper.setup(IntegrationTestBigLinkedList.java:455)
at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:143)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:799)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:347)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.base/java.security.AccessController.doPrivileged(Native Method)
at java.base/javax.security.auth.Subject.doAs(Subject.java:423)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1899)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
{code}
This task will add it back as part of the job configuration handled by hbase
internally in addHBaseDependencyJars
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Created] (HBASE-27055) Fix HBASE_TRACE_OPTS when adding opentelemetry agent
Tak-Lon (Stephen) Wu created HBASE-27055:
Summary: Fix HBASE_TRACE_OPTS when adding opentelemetry agent
Key: HBASE-27055
URL: https://issues.apache.org/jira/browse/HBASE-27055
Project: HBase
Issue Type: Bug
Components: tracing
Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3
Reporter: Tak-Lon (Stephen) Wu
it was a minor bug caused by HBASE-26363 with a typo in the following line
https://github.com/apache/hbase/blob/master/bin/hbase#L511
{code}
HBASE_OPTS="$HBASE_OPTS -javaagent:$agent_jar"
{code}
and it should be
{code}
HBASE_OPTS="$HBASE_TRACE_OPTS -javaagent:$agent_jar"
{code}
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Resolved] (HBASE-27033) Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
[ https://issues.apache.org/jira/browse/HBASE-27033?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27033. -- Hadoop Flags: Reviewed Release Note: see detail in HBASE-27013 Resolution: Fixed > Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" > to branch-2.4 > --- > > Key: HBASE-27033 > URL: https://issues.apache.org/jira/browse/HBASE-27033 > Project: HBase > Issue Type: Task >Affects Versions: 2.4.13 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.4.13 > > > Backport HBASE-27013 to branch-2.4, it's required because it's not a clean > backport. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-27013) Introduce read all bytes when using pread for prefetch
[
https://issues.apache.org/jira/browse/HBASE-27013?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-27013.
--
Fix Version/s: 2.5.0
2.6.0
3.0.0-alpha-3
Hadoop Flags: Reviewed
Release Note: Introduce optional flag hfile.pread.all.bytes.enabled for
pread that must read full bytes with the next block header, this is specially
helpful when users are running HBase with Blob storage like S3 and Azure Blob
storage.
Resolution: Fixed
> Introduce read all bytes when using pread for prefetch
> --
>
> Key: HBASE-27013
> URL: https://issues.apache.org/jira/browse/HBASE-27013
> Project: HBase
> Issue Type: Improvement
> Components: HFile, Performance
>Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> h2. Problem statement
> When prefetching HFiles from blob storage like S3 and use it with the storage
> implementation like S3A, we found there is a logical issue in HBase pread
> that causes the reading of the remote HFile aborts the input stream multiple
> times. This aborted stream and reopen slow down the reads and trigger many
> aborted bytes and waste time in recreating the connection especially when SSL
> is enabled.
> h2. ROOT CAUSE
> The root cause of above issue was due to
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> is reading an input stream that does not guarrentee to return the data block
> and the next block header as an option data to be cached.
> In the case of the input stream read short and when the input stream read
> passed the length of the necessary data block with few more bytes within the
> size of next block header, the
> [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257]
> returns to the caller without a cached the next block header. As a result,
> before HBase tries to read the next block,
> [HFileBlock#readBlockDataInternal|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664]
> in hbase tries to re-read the next block header from the input stream. Here,
> the reusable input stream has move the current position pointer ahead from
> the offset of the last read data block, when using with the [S3A
> implementation|https://github.com/apache/hadoop/blob/29401c820377d02a992eecde51083cf87f8e57af/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L339-L361],
> the input stream is then closed, aborted all the remaining bytes and reopen
> a new input stream at the offset of the last read data block .
> h2. How do we fix it?
> S3A is doing the right job that HBase is telling to move the offset from
> position A back to A - N, so there is not much thing we can do on how S3A
> handle the inputstream. meanwhile in the case of HDFS, this operation is fast.
> Such that, we should fix in HBase level, and try always to read datablock +
> next block header when we're using blob storage to avoid expensive draining
> the bytes in a stream and reopen the socket with the remote storage.
> h2. Draw back and discussion
> * A known drawback is, when we're at the last block, we will read extra
> length that should not be a header, and we still read that into the byte
> buffer array. the size should be always 33 bytes, and it should not a big
> issue in data correctness because the trailer will tell when the last
> datablock should end. And we just waste a 33 byte read and that data is not
> being used.
> * I don't know if we can use HFileStreamReader but that will change the
> Prefetch logic a lot, such that this minimum change should be the best.
> h2. initial result
> We use YCSB 1 billion records data, and we enable prefetch for the userable.
> the collected the S3A metrics of {{stream_read_bytes_discarded_in_abort}} to
> compare the solution, each region server have abort ~290 GB data to be
> prefetch to bucketcache.
> * before the change, we have a total of 4235973338472 bytes (~4235GB) has
> been aborted on a sample region server for about 290GB data.
> ** the overall time was about 45 ~ 60 mins
>
> {code}
> % grep "stream_read_bytes_discarded_in_abort"
> ~/prefetch-result/prefetch-s3a-jmx-metrics.json | grep -wv
> "stream_read_bytes_discarded_in_abort\":0,"
>
[jira] [Created] (HBASE-27033) Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4
Tak-Lon (Stephen) Wu created HBASE-27033: Summary: Backport "HBASE-27013 Introduce read all bytes when using pread for prefetch" to branch-2.4 Key: HBASE-27033 URL: https://issues.apache.org/jira/browse/HBASE-27033 Project: HBase Issue Type: Task Affects Versions: 2.4.13 Reporter: Tak-Lon (Stephen) Wu Fix For: 2.4.13 Backport HBASE-27013 to branch-2.4, it's required because it's not a clean backport. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Created] (HBASE-27013) Introduce read all bytes when using pread for Prefetch
Tak-Lon (Stephen) Wu created HBASE-27013: Summary: Introduce read all bytes when using pread for Prefetch Key: HBASE-27013 URL: https://issues.apache.org/jira/browse/HBASE-27013 Project: HBase Issue Type: Improvement Components: HFile, Performance Affects Versions: 2.5.0, 2.6.0, 3.0.0-alpha-3, 2.4.13 Reporter: Tak-Lon (Stephen) Wu h2. Problem statement When prefetching HFiles from blob storage like S3 and use it with the storage implementation like S3A, we found there is a logical issue in HBase pread that causes the reading of the remote HFile aborts the input stream multiple times. This aborted stream and reopen slow down the reads and trigger many aborted bytes and waste time in recreating the connection especially when SSL is enabled. h2. ROOT CAUSE The root cause of above issue was due to [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257] is reading an input stream that does not guarrentee to return the data block and the next block header as an option data to be cached. In the case of the input stream read short and we passed the length of the necessary data block plus few bytes within the size of next block header, [BlockIOUtils#preadWithExtra|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-common/src/main/java/org/apache/hadoop/hbase/io/util/BlockIOUtils.java#L214-L257] returns to the caller. As a result, when we're trying to read the next block, [HFileBlock#|https://github.com/apache/hbase/blob/9c8c9e7fbf8005ea89fa9b13d6d063b9f0240443/hbase-server/src/main/java/org/apache/hadoop/hbase/io/hfile/HFileBlock.java#L1648-L1664] in hbase will try to re-read the header from the input stream, but because the reusable input stream has move the current position pointer ahead from the offset of the last read data block, with the [S3A implementation|https://github.com/apache/hadoop/blob/29401c820377d02a992eecde51083cf87f8e57af/hadoop-tools/hadoop-aws/src/main/java/org/apache/hadoop/fs/s3a/S3AInputStream.java#L339-L361], the input stream is then closed, aborted all the remaining bytes and reopen a new input stream at the offset of the last read data block . h2. How do we fix it? S3A is doing the right job that HBase is telling to move the offset from position A back to A - N, so there is not much thing we can do on how S3A handle the inputstream. meanwhile in the case of HDFS, this operation is fast. Such that, we should fix in HBase level, and try always to read datablock + next block header when we're using blob storage to avoid expensive draining the bytes in a stream and reopen the socket with the remote storage. h2. Draw back and discussion * A known drawback is, when we're at the last block, we will read extra length that should not be a header, and we still read that into the byte buffer array. the size should be always 33 bytes, and it should not a big issue in data correctness because the trailer will tell when the last datablock should end. And we just waste a 33 byte read and that data is not being used. * I don't know if we can use HFileStreamReader but that will change the Prefetch logic a lot, such that this minimum change should be the best. -- This message was sent by Atlassian Jira (v8.20.7#820007)
[jira] [Resolved] (HBASE-26882) Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26882?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26882. -- Fix Version/s: 2.6.0 2.4.12 Hadoop Flags: Reviewed Resolution: Fixed > Backport "HBASE-26810 Add dynamic configuration support for system > coprocessors" to branch-2 > > > Key: HBASE-26882 > URL: https://issues.apache.org/jira/browse/HBASE-26882 > Project: HBase > Issue Type: Task > Components: Coprocessors, master, regionserver >Affects Versions: 2.5.0, 2.6.0, 2.4.12 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.6.0, 2.4.12 > > > Backport HBASE-26810 to streams of branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26810) Add dynamic configuration support for system coprocessors
[ https://issues.apache.org/jira/browse/HBASE-26810?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26810. -- Hadoop Flags: Reviewed Release Note: Now dynamic configuration, configurign hbase-site and then call update_all_config, can also applied for the following properties. hbase.coprocessor.master.classes hbase.coprocessor.region.classes hbase.coprocessor.regionserver.classes hbase.coprocessor.user.region.classes Resolution: Fixed committed as https://github.com/apache/hbase/commit/4ac600b04b7537d1bb171ba1501ad845dbb37fdb , create a backport request HBASE-26882 for branch-2 > Add dynamic configuration support for system coprocessors > - > > Key: HBASE-26810 > URL: https://issues.apache.org/jira/browse/HBASE-26810 > Project: HBase > Issue Type: Bug > Components: conf, Coprocessors >Affects Versions: 3.0.0-alpha-3 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > [Dynamic Configuraiton|https://hbase.apache.org/book.html#dyn_config] is very > helpful that the operator can keep the JVM for regionserver or master running > without restarting it. > With this feature, this task aims to extend the scope of on demend > configuration to system coprocessors including REGION, USER REGION, > REGIONSERVER, MASTER such that we could save time on rolling restart with a > quicker update. > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26882) Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2
Tak-Lon (Stephen) Wu created HBASE-26882: Summary: Backport "HBASE-26810 Add dynamic configuration support for system coprocessors" to branch-2 Key: HBASE-26882 URL: https://issues.apache.org/jira/browse/HBASE-26882 Project: HBase Issue Type: Task Components: Coprocessors, master, regionserver Affects Versions: 2.5.0, 2.6.0, 2.4.12 Reporter: Tak-Lon (Stephen) Wu Backport HBASE-26810 to streams of branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26841) Replacing log4j with reload4j for hbase-connector
Tak-Lon (Stephen) Wu created HBASE-26841: Summary: Replacing log4j with reload4j for hbase-connector Key: HBASE-26841 URL: https://issues.apache.org/jira/browse/HBASE-26841 Project: HBase Issue Type: Task Components: hbase-connectors Affects Versions: 1.0.0 Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu Fix For: 1.1.0 As a result of the [discussion thread of replacing log4j with reload4j|https://lists.apache.org/thread/kfmjg6zmvdjqcwolj0oh634nzv42y806] and HBASE-26691, we have replaced log4j with reload4j in branch-2. Where hbase-connector as a part of the community, this JIRA propose to align with the active hbase release and replace log4j with reload4j before the next release of hbase-connector -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26804) Missing opentelemetry agent in hadoop-two-compat.xml
[ https://issues.apache.org/jira/browse/HBASE-26804?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26804. -- Hadoop Flags: Reviewed Resolution: Fixed > Missing opentelemetry agent in hadoop-two-compat.xml > > > Key: HBASE-26804 > URL: https://issues.apache.org/jira/browse/HBASE-26804 > Project: HBase > Issue Type: Bug > Components: tracing >Affects Versions: 2.5.0, 2.6.0 >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Blocker > Fix For: 2.5.0, 2.6.0 > > > We do not have hadoop-two-compat.xml on master, this is probably why we > missed this file when backporting to branch-2. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26810) Add dynamic configuration support for system coprocessors
Tak-Lon (Stephen) Wu created HBASE-26810: Summary: Add dynamic configuration support for system coprocessors Key: HBASE-26810 URL: https://issues.apache.org/jira/browse/HBASE-26810 Project: HBase Issue Type: Bug Components: conf, Coprocessors Affects Versions: 3.0.0-alpha-3 Reporter: Tak-Lon (Stephen) Wu [Dynamic Configuraiton|https://hbase.apache.org/book.html#dyn_config] is very helpful that the operator can keep the JVM for regionserver or master running without restarting it. With this feature, this task aims to extend the scope of on demend configuration to system coprocessors including REGION, USER REGION, REGIONSERVER, MASTER such that we could save time on rolling restart with a quicker update. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-22338) LICENSE file only contains Apache 2.0
[ https://issues.apache.org/jira/browse/HBASE-22338?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-22338. -- Fix Version/s: hbase-connectors-1.1.0 (was: hbase-connectors-1.0.1) Hadoop Flags: Reviewed Resolution: Fixed [036729dea0d1041e957f0349254f4ea1cf7cfba9|https://github.com/apache/hbase-connectors/commit/036729dea0d1041e957f0349254f4ea1cf7cfba9] pushed, resolving > LICENSE file only contains Apache 2.0 > - > > Key: HBASE-22338 > URL: https://issues.apache.org/jira/browse/HBASE-22338 > Project: HBase > Issue Type: Bug > Components: hbase-connectors >Affects Versions: connector-1.0.0 >Reporter: Peter Somogyi >Assignee: Tak-Lon (Stephen) Wu >Priority: Critical > Fix For: hbase-connectors-1.1.0 > > Attachments: NOTICE.aggregate-no-build-year, > hbase-connectors-dependency.html > > > LICENSE.md file has only Apache 2.0 licenses but we package dependencies that > use different ones. For example jcodings uses MIT. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26729) Backport "HBASE-26714 Introduce path configuration for system coprocessors" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26729. -- Fix Version/s: 2.5.0 2.4.10 Hadoop Flags: Reviewed Resolution: Fixed pushed to branch-2.4 https://github.com/apache/hbase/commit/b8db5ac24c16fe9b4216cb5f6fef6df91977823a and branch-2 https://github.com/apache/hbase/commit/f637a60bdd66d2408be0f4fdbb3ced66f0958561 > Backport "HBASE-26714 Introduce path configuration for system coprocessors" > to branch-2 > --- > > Key: HBASE-26729 > URL: https://issues.apache.org/jira/browse/HBASE-26729 > Project: HBase > Issue Type: Bug > Components: Coprocessors >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0, 2.4.10 > > > backport HBASE-26714 to branch-2, mainly it's due to the unclear cherry pick > of the change of HBaseCommonTestUtility to HBaseCommonTestingUtility -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26714) Introduce path configuration for system coprocessors
[
https://issues.apache.org/jira/browse/HBASE-26714?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26714.
--
Release Note: After HBASE-26714, users of hbase can now set system
coprocessor with a path of the JAR file other than the files in default
classpath. E.g. when setting
hbase.coprocessor.region.classes/hbase.coprocessor.regionserver.classes/hbase.coprocessor.user.region.classes/hbase.coprocessor.master.classes,
user can set a comma-separated list with the format of
className|priority|path, where the path could be a file path of the supported
filesystem.
Tags: coprocessors
Resolution: Fixed
we will have the backport to branch-2 in HBASE-26729
> Introduce path configuration for system coprocessors
>
>
> Key: HBASE-26714
> URL: https://issues.apache.org/jira/browse/HBASE-26714
> Project: HBase
> Issue Type: Task
> Components: Coprocessors
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Currently when region server starts, system coprocessors are normally loaded
> as part of the local classpath and the jar is stored in the local storage; in
> short words, Operator would have to copy the jar to each host (or a shared
> file system) and construct it as part of HBASE_CLASSPATH.
> although operator may have been doing the presetup of copying jar and making
> it available to the HBASE_CLASSPATH without any issue, it could be helpful if
> we provide an alternative method and centralize this configuration in
> hbase-site similar to the support of table-level coprocessor, e.g.
> configuring {{hbase.coprocessor.region.classes}} with the local/remote path
> along with the classname with a specific path token in the form of
> {{className|priority|path}}.
> Similarly in HBASE-23710, it provided the priority configuration via
> hbase-site that this new improvement aligns with the same purpose, and
> further help to simplify the deployment of System Coprocessors to e.g. hdfs
> or supported cloud storage.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26729) Backport "HBASE-26714 Introduce path configuration for system coprocessors" to branch-2
Tak-Lon (Stephen) Wu created HBASE-26729: Summary: Backport "HBASE-26714 Introduce path configuration for system coprocessors" to branch-2 Key: HBASE-26729 URL: https://issues.apache.org/jira/browse/HBASE-26729 Project: HBase Issue Type: Bug Components: Coprocessors Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu backport HBASE-26714 to branch-2, mainly it's due to the unclear cherry pick of the change of HBaseCommonTestUtility to HBaseCommonTestingUtility -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26714) Introduce path configuration for system coprocessors
Tak-Lon (Stephen) Wu created HBASE-26714:
Summary: Introduce path configuration for system coprocessors
Key: HBASE-26714
URL: https://issues.apache.org/jira/browse/HBASE-26714
Project: HBase
Issue Type: Task
Components: Coprocessors
Affects Versions: 2.5.0, 3.0.0-alpha-3
Reporter: Tak-Lon (Stephen) Wu
Currently when region server starts, system coprocessors are normally loaded as
part of the local classpath and the jar is stored in the local storage; in
short words, Operator would have to copy the jar to each host (or a shared file
system) and construct it as part of HBASE_CLASSPATH.
although operator may have been doing the presetup of copying jar and making it
available to the HBASE_CLASSPATH without any issue, it could be helpful if we
provide an alternative method and centralize this configuration in hbase-site
similar to the support of table-level coprocessor, e.g. configuring
{{hbase.coprocessor.region.classes}} with the local/remote path along with the
classname with a specific path token in the form of {{className|priority|path}}.
Similarly in HBASE-23710, it provided the priority configuration via hbase-site
that this new improvement aligns with the same purpose, and further help to
simplify the deployment of System Coprocessors to e.g. hdfs or supported cloud
storage.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26664) HBASE-26664 hbase-connector upgrades extra-enforcer-rules to 1.5.1
[
https://issues.apache.org/jira/browse/HBASE-26664?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26664.
--
Hadoop Flags: Reviewed
Resolution: Fixed
> HBASE-26664 hbase-connector upgrades extra-enforcer-rules to 1.5.1
> --
>
> Key: HBASE-26664
> URL: https://issues.apache.org/jira/browse/HBASE-26664
> Project: HBase
> Issue Type: Task
> Components: hbase-connectors
>Affects Versions: hbase-connectors-1.1.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: hbase-connectors-1.1.0
>
>
> hbase-connector fails when building with -Prelease after
> maven-enforcer-plugin upgraded to 3.0.0 from 3.0.0-M3 in HBASE-26534.
> We upgraded maven-enforcer-plugin from 3.0.0-M3 to 3.0.0, but then we hit the
> problem of `extra-enforcer-rules` has different as mentioned in
> https://github.com/mojohaus/extra-enforcer-rules/issues/131#issuecomment-890329715
> {code}
> $ mvn clean install -Prelease
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce
> (enforce-maven-version) on project hbase-connectors: Unable to parse
> configuration of mojo
> org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce for parameter
> enforceBytecodeVersion: Cannot create instance of class
> org.apache.maven.plugins.enforcer.EnforceBytecodeVersion:
> org/apache/maven/shared/dependency/tree/DependencyTreeBuilderException:
> org.apache.maven.shared.dependency.tree.DependencyTreeBuilderException ->
> [Help 1]
> {code}
> https://github.com/mojohaus/extra-enforcer-rules/commit/d380fe56a7317a237bbce5b94fbab7f4581d3d1b
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26664) hbase-connector fails when building with -Prelease
Tak-Lon (Stephen) Wu created HBASE-26664:
Summary: hbase-connector fails when building with -Prelease
Key: HBASE-26664
URL: https://issues.apache.org/jira/browse/HBASE-26664
Project: HBase
Issue Type: Task
Components: hbase-connectors
Affects Versions: hbase-connectors-1.1.0
Reporter: Tak-Lon (Stephen) Wu
Fix For: hbase-connectors-1.1.0
after HBASE-26534, we upgraded maven-enforcer-plugin from 3.0.0-M3 to 3.0.0,
but then we hit the problem of `extra-enforcer-rules` has different as
mentioned in
https://github.com/mojohaus/extra-enforcer-rules/issues/131#issuecomment-890329715
{code}
$ mvn clean install -Prelease
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce
(enforce-maven-version) on project hbase-connectors: Unable to parse
configuration of mojo
org.apache.maven.plugins:maven-enforcer-plugin:3.0.0:enforce for parameter
enforceBytecodeVersion: Cannot create instance of class
org.apache.maven.plugins.enforcer.EnforceBytecodeVersion:
org/apache/maven/shared/dependency/tree/DependencyTreeBuilderException:
org.apache.maven.shared.dependency.tree.DependencyTreeBuilderException -> [Help
1]
{code}
https://github.com/mojohaus/extra-enforcer-rules/commit/d380fe56a7317a237bbce5b94fbab7f4581d3d1b
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-24356) Move hbase-connectors to hbase-thirdparty-3.3.0
[ https://issues.apache.org/jira/browse/HBASE-24356?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-24356. -- Resolution: Duplicate HBASE-26534 has the changes that we needs in this PR, and [~lucacanali] has merged the required changes from this JIRA, resolving as duplicates > Move hbase-connectors to hbase-thirdparty-3.3.0 > --- > > Key: HBASE-24356 > URL: https://issues.apache.org/jira/browse/HBASE-24356 > Project: HBase > Issue Type: Task > Components: hbase-connectors >Affects Versions: hbase-connectors-1.0.1 >Reporter: Peter Somogyi >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: hbase-connectors-1.1.0 > > -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26534) Update dependencies in hbase-connectors: HBase version to 2.4.8, and make Hadoop 3 and Spark 3 defaults
[ https://issues.apache.org/jira/browse/HBASE-26534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26534. -- Fix Version/s: 1.1.0 Hadoop Flags: Reviewed Release Note: HBASE-26534 upgrades hbase to 2.4.9, spark to 3.1.2, and hadoop to 3.2.0. Also It builds with spark3 with scala-2.12 and hadoop3 profile as default option. Assignee: Luca Canali Resolution: Fixed Thanks [~lucacanali] for your contribution > Update dependencies in hbase-connectors: HBase version to 2.4.8, and make > Hadoop 3 and Spark 3 defaults > --- > > Key: HBASE-26534 > URL: https://issues.apache.org/jira/browse/HBASE-26534 > Project: HBase > Issue Type: Improvement >Reporter: Luca Canali >Assignee: Luca Canali >Priority: Minor > Fix For: 1.1.0 > > > This proposes to bump up the default versions used by hbase-connectors to > recent versions, in particular for HBase, Spark, Hadoop. > HBase is upgraded to 2.4.8, Spark to 3.1.2, Hadoop to 3.2.0. > This proposes also to make Spark 3 and Hadoop 3 default version (as opposed > to Spark 2 and Hadoop 2). -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
[ https://issues.apache.org/jira/browse/HBASE-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26530. -- Hadoop Flags: Reviewed Resolution: Fixed > Backport HBASE-26524 Support remove coprocessor by class name via alter table > command > -- > > Key: HBASE-26530 > URL: https://issues.apache.org/jira/browse/HBASE-26530 > Project: HBase > Issue Type: Task > Components: Coprocessors, shell >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.6.0 > > > porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Hadoop Flags: Incompatible change,Reviewed
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: incompatible, incompatibleChange
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu reopened HBASE-26524:
--
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: incompatible, incompatibleChange
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26529) Document HBASE-26524 to section of Dynamic Unloading
[
https://issues.apache.org/jira/browse/HBASE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26529.
--
Fix Version/s: 3.0.0-alpha-2
Hadoop Flags: Reviewed
Release Note:
added the basic command usage of `table_remove_coprocessor` to the definition
guide at src/main/asciidoc/_chapters/cp.adoc.
hbase> alter 'users', METHOD => 'table_remove_coprocessor', CLASSNAME =>
'org.myname.hbase.Coprocessor.RegionObserverExample'
Resolution: Fixed
> Document HBASE-26524 to section of Dynamic Unloading
>
>
> Key: HBASE-26529
> URL: https://issues.apache.org/jira/browse/HBASE-26529
> Project: HBase
> Issue Type: Task
> Components: Coprocessors, shell
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0-alpha-2
>
>
> HBASE-26524 has merged, and we need to update the documentation such that
> operator knows there is a method of {{table_remove_coprocessor}} for
> {{alter}} command
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu reopened HBASE-26524:
--
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Implemented
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
Tak-Lon (Stephen) Wu created HBASE-26530: Summary: Backport HBASE-26524 Support remove coprocessor by class name via alter table command Key: HBASE-26530 URL: https://issues.apache.org/jira/browse/HBASE-26530 Project: HBase Issue Type: Task Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Created] (HBASE-26529) Add document for HBASE-26524 to book.html
Tak-Lon (Stephen) Wu created HBASE-26529:
Summary: Add document for HBASE-26524 to book.html
Key: HBASE-26529
URL: https://issues.apache.org/jira/browse/HBASE-26529
Project: HBase
Issue Type: Task
Components: Coprocessors, shell
Reporter: Tak-Lon (Stephen) Wu
HBASE-26524 has merged, and we need to update the documentation such that
operator knows there is a method of {{table_remove_coprocessor}} for {{alter}}
command
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26524) Support remove coprocessor by class name via alter table command
Tak-Lon (Stephen) Wu created HBASE-26524:
Summary: Support remove coprocessor by class name via alter table
command
Key: HBASE-26524
URL: https://issues.apache.org/jira/browse/HBASE-26524
Project: HBase
Issue Type: Improvement
Components: Coprocessors, shell
Affects Versions: 2.5.0, 3.0.0-alpha-2
Reporter: Tak-Lon (Stephen) Wu
With the shell, when operator wants to remove a table coprocessor, the flow is
to
1. first use {{decs}} find the the mapping of coprocessor$# e.g. coprocessor$1,
where # is the ordered number when the coprocessor was internally added to the
table attribute
2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
value that include the unique class name.
This task is to simplify the flow if the operator know exactly the class name
of the added coprocessor, and create a new sub-method to {{alter}}, such that
operator can do it only with the class name.
NOTE that this logic has been added behind the scenes at
[TableDescriptorBuilder#removeCoprocessor](https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565)
for removing `ConstraintProcessor` , and we are just exposing this logic with
a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-24833) Bootstrap should not delete the META table directory if it's not partial
[
https://issues.apache.org/jira/browse/HBASE-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-24833.
--
Resolution: Fixed
> Bootstrap should not delete the META table directory if it's not partial
>
>
> Key: HBASE-24833
> URL: https://issues.apache.org/jira/browse/HBASE-24833
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.3.1, 2.3.3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> this issues were discussed in
> [PR#2113|https://github.com/apache/hbase/pull/2113] as part of HBASE-24286,
> and it is a dependencies before we solve HBASE-24286.
> The changes were introduced in [HBASE-24471
> |https://github.com/apache/hbase/commit/4d5efec76718032a1e55024fd5133409e4be3cb8#diff-21659161b1393e6632730dcbea205fd8R70-R89]
> that partial meta was introduced and `partial` was defined as
> InitMetaProcedure did not succeed and INIT_META_ASSIGN_META was not completed.
> {code:java}
> private static void writeFsLayout(Path rootDir, Configuration conf) throws
> IOException {
>LOG.info("BOOTSTRAP: creating hbase:meta region");
>FileSystem fs = rootDir.getFileSystem(conf);
>Path tableDir = CommonFSUtils.getTableDir(rootDir,
> TableName.META_TABLE_NAME);
>if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
> LOG.warn("Can not delete partial created meta table, continue...");
>}
> {code}
> however, in the cloud use case where HFiles store on S3, WALs store on HDFS,
> ZK data are stored within the cluster, this partial meta becomes a block when
> cluster recreate on existing HFiles; Here, Zk data and WALs cannot be
> retained (HDFS was associated with cloud instance and was terminated
> together) when cluster recreates on the flushed HFiles, and existing meta are
> always considered as partial and deleted in `INIT_META_WRITE_FS_LAYOUT`
> during bootstrap. As a result, the recreate cluster starts with a empty meta
> table, either the cluster hangs during the master initialization (branch-2)
> because table states of namespace table cannot be assigned, or starts as a
> fresh cluster without any region assigned and table opens (may need HBCK to
> rebuild the meta).
> Potential solution suggested by Anoop
> {quote}In case of HM start and the bootstrap we create the ClusterID and
> write to FS and then to zk and then create the META table FS layout. So in a
> cluster recreate, we will see clusterID is there in FS and also the META FS
> layout but no clusterID in zk. Ya seems we can use this as indication for
> cluster recreate over existing data. In HM start, this is some thing we need
> to check at 1st itself and track. If this mode is true, later when (if) we do
> INIT_META_WRITE_FS_LAYOUT , we should not delete the META dir. As part of the
> Bootstrap when we write that proc to MasterProcWal, we can include this mode
> (boolean) info also. This is a protobuf message anyways. So even if this HM
> got killed and restarted (at a point where the clusterId was written to zk
> but the Meta FS layout part was not reached) we can use the info added as
> part of the bootstrap wal entry and make sure NOT to delete the meta dir.
> {quote}
> In this JIRA, we're going to fix the `partial` definition when we found
> cluster ID was stored in HFiles but ZK were deleted or fresh on cluster
> creates.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26299) Fix TestHTableTracing.testTableClose for nightly build of branch-2
[
https://issues.apache.org/jira/browse/HBASE-26299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26299.
--
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix TestHTableTracing.testTableClose for nightly build of branch-2
> --
>
> Key: HBASE-26299
> URL: https://issues.apache.org/jira/browse/HBASE-26299
> Project: HBase
> Issue Type: Bug
> Components: test, tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> after merging HBASE-26141, sometime isn't right with the last testTableClose
> when we close the table and the connection, need to figure out why it's not
> working in the unit test with the jdk8 and hadoop 3 profile
> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/
> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/352/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/
> {code}
> [ERROR] org.apache.hadoop.hbase.client.TestHTableTracing.testTableClose Time
> elapsed: 0.001 s <<< ERROR!
> java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been
> called. GlobalOpenTelemetry.set must be called only once before any calls to
> GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use
> OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation
> set to cause of this exception.
> at
> io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:83)
> at
> io.opentelemetry.sdk.testing.junit4.OpenTelemetryRule.before(OpenTelemetryRule.java:95)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.Throwable
> at
> io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:91)
> at
> io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:61)
> at
> io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:110)
> at
> org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:71)
> at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:95)
> at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:78)
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$trace$1(TraceUtil.java:176)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:180)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:176)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.close(ConnectionImplementation.java:2110)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.finalize(ConnectionImplementation.java:2149)
> at java.lang.System$2.invokeFinalize(System.java:1273)
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
> at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26299) Fix TestHTableTracing.testTableClose for nightly build of branch-2
Tak-Lon (Stephen) Wu created HBASE-26299:
Summary: Fix TestHTableTracing.testTableClose for nightly build of
branch-2
Key: HBASE-26299
URL: https://issues.apache.org/jira/browse/HBASE-26299
Project: HBase
Issue Type: Bug
Components: test, tracing
Affects Versions: 2.5.0
Reporter: Tak-Lon (Stephen) Wu
sometime isn't right with the last testTableClose when we close the table and
the connection, need to figure out why it's not working in the unit test.
{code}
[ERROR] org.apache.hadoop.hbase.client.TestHTableTracing.testTableClose Time
elapsed: 0.001 s <<< ERROR!
java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been
called. GlobalOpenTelemetry.set must be called only once before any calls to
GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use
OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set
to cause of this exception.
at
io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:83)
at
io.opentelemetry.sdk.testing.junit4.OpenTelemetryRule.before(OpenTelemetryRule.java:95)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Throwable
at
io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:91)
at
io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:61)
at
io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:110)
at
org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:71)
at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:95)
at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:78)
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$trace$1(TraceUtil.java:176)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:180)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:176)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.close(ConnectionImplementation.java:2110)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.finalize(ConnectionImplementation.java:2149)
at java.lang.System$2.invokeFinalize(System.java:1273)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26141) Add tracing support for HTable and sync connection on branch-2
[ https://issues.apache.org/jira/browse/HBASE-26141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26141. -- Hadoop Flags: Reviewed Resolution: Fixed > Add tracing support for HTable and sync connection on branch-2 > -- > > Key: HBASE-26141 > URL: https://issues.apache.org/jira/browse/HBASE-26141 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > Split from HBASE-25853, the original change of HBASE-22120 does not include > tracing support for sync client, if we decide to support it in branch-2, we > will need to come up our own solution. > this change should cover mainly the classes of HTable, > ConnectionImplementation, and HRegionLocator > * RPC and IPC tracing supported by HBASE-26125 > * WAL tracing supported by HBASE-26131 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26274) Create an option to reintroduce BlockCache to mapreduce job
[
https://issues.apache.org/jira/browse/HBASE-26274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26274.
--
Resolution: Fixed
> Create an option to reintroduce BlockCache to mapreduce job
> ---
>
> Key: HBASE-26274
> URL: https://issues.apache.org/jira/browse/HBASE-26274
> Project: HBase
> Issue Type: Bug
> Components: BlockCache, HFile, mapreduce
>Affects Versions: 3.0.0-alpha-1, 2.4.6
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0, 2.3.7, 2.4.7, 3.0.0-alpha-1
>
>
> In HBASE-21498 (see [this
> commit|https://github.com/apache/hbase/commit/27a0f205c52f83fe7500ee2ffc6cf6582f565a63#diff-8a3e39e6df1afe47811fc17702da598fe0d80496d66e579bea4bd224c6d8da03R218],
> it change the behavior that only region server can initialize on-heap
> BlockCache/LruBlockCache, this should be the right change for HMaster.
> Other downstream dependency that uses getScanner from a file-based region and
> read HStore/HFile lost the BlockCache for caching INDEX/LEAF_INDEX (at least
> still a problem with HBase-2.4) after this change (it worked before) and
> caused performance impact with 2x slower.
> One way to bring back the performance is to allow non-RS and non-HMaster can
> use a compact version of blockcache with smaller memory and less hbase
> internal configuration.
> Or if we can find a way to cache or skip reading the same {{LEAF_INDEX}} when
> scanning the DATA block with HFile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26274) Create an option to reintroduce BlockCache to mapreduce job
Tak-Lon (Stephen) Wu created HBASE-26274:
Summary: Create an option to reintroduce BlockCache to mapreduce
job
Key: HBASE-26274
URL: https://issues.apache.org/jira/browse/HBASE-26274
Project: HBase
Issue Type: Bug
Components: BlockCache, HFile, mapreduce
Affects Versions: 2.4.6, 3.0.0-alpha-1
Reporter: Tak-Lon (Stephen) Wu
In HBASE-21498 (see [this
commit|https://github.com/apache/hbase/commit/27a0f205c52f83fe7500ee2ffc6cf6582f565a63#diff-8a3e39e6df1afe47811fc17702da598fe0d80496d66e579bea4bd224c6d8da03R218],
it change the behavior that only region server can initialize on-heap
BlockCache/LruBlockCache, this should be the right change for HMaster.
Other downstream dependency that uses getScanner from a file-based region and
read HStore/HFile lost the BlockCache for caching INDEX/LEAF_INDEX (at least
still a problem with HBase-2.4) after this change (it worked before) and caused
performance impact with 2x slower.
One way to bring back the performance is to allow non-RS and non-HMaster can
use a compact version of blockcache with smaller memory and less hbase internal
configuration.
Or if we can find a way to cache or skip reading the same {{LEAF_INDEX}} when
scanning the DATA block with HFile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26273) TableSnapshotInputFormat/TableSnapshotInputFormatImpl should use ReadType.STREAM for scanning HFiles
Tak-Lon (Stephen) Wu created HBASE-26273:
Summary: TableSnapshotInputFormat/TableSnapshotInputFormatImpl
should use ReadType.STREAM for scanning HFiles
Key: HBASE-26273
URL: https://issues.apache.org/jira/browse/HBASE-26273
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 2.4.6, 3.0.0-alpha-1
Reporter: Tak-Lon (Stephen) Wu
After the change in HBASE-17917 that use PREAD ({{ReadType.DEFAULT}}) for all
user scan, the behavior of TableSnapshotInputFormat changed from STREAM to
PREAD.
TableSnapshotInputFormat is supposed to be use with a YARN/MR or other batch
engine that should read the entire HFile in the container/executor, with
default always to PREAD, the number of connection to HDFS surges and has an
side-effect on the overall performance.
The goal of this change is to make any downstream using
TableSnapshotInputFormat with STREAM scan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25853) Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
[
https://issues.apache.org/jira/browse/HBASE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-25853.
--
Resolution: Fixed
https://github.com/apache/hbase/pull/3637 merged into branch-2
> Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
>
>
> Key: HBASE-25853
> URL: https://issues.apache.org/jira/browse/HBASE-25853
> Project: HBase
> Issue Type: Task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Duo Zhang
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> list of commits from upstream/master need to be backported
> {code}
> HBASE-25373
> https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef
>
> HBASE-25401
> https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04
>
> HBASE-25424
> https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d
>
> HBASE-23898
> https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59
>
> HBASE-25454
> https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93
> HBASE-25481
> https://github.com/apache/hbase/commit/ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
> HBASE-25455
> https://github.com/apache/hbase/commit/03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
> HBASE-25484
> https://github.com/apache/hbase/commit/2be2c63f0d3917a243b74af9754cbfc805b858d1
> HBASE-25535
> https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3
> HBASE-25591
> https://github.com/apache/hbase/commit/f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
> HBASE-25617
> https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294
> HBASE-25616
> https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99
> HBASE-25723
> https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea
> HBASE-25732
> https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071
> HBASE-25733
> https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673
> HBASE-23762
> https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238
> HBASE-25778
> https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26241) Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin
Tak-Lon (Stephen) Wu created HBASE-26241: Summary: Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin Key: HBASE-26241 URL: https://issues.apache.org/jira/browse/HBASE-26241 Project: HBase Issue Type: Task Affects Versions: 3.0.0-alpha-1, 2.5.0 Reporter: Tak-Lon (Stephen) Wu This task/issue was being discussed [in the vote of HBASE-25853|https://lists.apache.org/thread.html/rb36f0b6b179ee5f42cff7bb60de2797f69c450752c993f0b1e0f6118%40%3Cdev.hbase.apache.org%3E] bq. I've not seen anyone report their success at integrating these changes into a telemetry system and showing end-to-end tracing of a service call. As I recall from the last 2.5 discuss thread, we wanted this to be the banner feature for the minor release. Since we don't have this end-to-end confirmation, do we have someone who's volunteered to demonstrate that final proof-of-integration in a timely manner? Perhaps that someone is keen to be release manager for 2.5. What a `service call` mean ? (pending discussion on above email exchange or on this JIRA) e.g. for RPCclient/AbstractRpcClient, it trace at the AbstractRpcClient#callMethod level with a span of `RpcClient.callMethod` . or whenever a connection is closed, it will emit `AsyncConnection.close` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26168) Backport HBASE-25811 for fixing nightly tests with error of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
[
https://issues.apache.org/jira/browse/HBASE-26168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26168.
--
Hadoop Flags: Reviewed
Resolution: Fixed
> Backport HBASE-25811 for fixing nightly tests with error of
> `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
>
>
> Key: HBASE-26168
> URL: https://issues.apache.org/jira/browse/HBASE-26168
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> see nightly error
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/313/artifact/output-integration/hadoop-2/mr-importtsv.err/*view*/]
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> io/opentelemetry/api/GlobalOpenTelemetry
> at org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:33)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:398)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:589)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:45761)
> at
> org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:50)
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:372)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
> ... 32 more
> Caused by: java.lang.ClassNotFoundException:
> io.opentelemetry.api.GlobalOpenTelemetry
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 43 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26140) Backport HBASE-25778 "The tracinig implementation for AsyncConnectionImpl.getHbck is incorrect" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26140.
--
Hadoop Flags: Reviewed
Resolution: Fixed
https://github.com/apache/hbase/commit/d80763268621590b0701ec3fa2518fab6834d1b0
pushed
> Backport HBASE-25778 "The tracinig implementation for
> AsyncConnectionImpl.getHbck is incorrect" to branch-2
> ---
>
> Key: HBASE-26140
> URL: https://issues.apache.org/jira/browse/HBASE-26140
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 17/17 commits of HBASE-22120, original commit
> [{{f36e153}}|https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26226) Make RpcConnection with transient/non-Singleton SaslClientAuthenticationProviders with repsecting the passing configuration object
Tak-Lon (Stephen) Wu created HBASE-26226:
Summary: Make RpcConnection with transient/non-Singleton
SaslClientAuthenticationProviders with repsecting the passing configuration
object
Key: HBASE-26226
URL: https://issues.apache.org/jira/browse/HBASE-26226
Project: HBase
Issue Type: Improvement
Components: Client, rpc
Reporter: Tak-Lon (Stephen) Wu
When a RPC connection was created from any table connection (e.g. via
{{ConnectionFactory.createConnection(conf)}}), the
[SaslClientAuthenticationProviders|(https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcConnection.java#L99-L101]
that matches different authentication selector per-user/TOKEN as the
[singleton
object|https://github.com/apache/hbase/blob/735bcf85e9b59a71981babe2f5da51978d61d8d3/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/SaslClientAuthenticationProviders.java#L72-L83]
and reuses for any followup connection creation.
If user want to create another connection with a different configuration key
that the authentication selector may use, e.g. [if one want to switch between
secure and non-secure
connection|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/BuiltInProviderSelector.java#L109-L111],
it cannot be done because of the cached singleton instance of
AuthenticationProviders.
the goal of this JIRA is to allow the RpcConnection and AuthenticationProviders
align with whatever passing configuration object
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26139) Backport HBASE-23762 "Add documentation on how to enable and view tracing with OpenTelemetry" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26139.
--
Hadoop Flags: Reviewed
Resolution: Fixed
[https://github.com/apache/hbase/commit/c96746736fa868d5a8859b501b6ab11e97f1a674]
pushed
> Backport HBASE-23762 "Add documentation on how to enable and view tracing
> with OpenTelemetry" to branch-2
> -
>
> Key: HBASE-26139
> URL: https://issues.apache.org/jira/browse/HBASE-26139
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 16/17 commits of HBASE-22120, original commit
> [{{be4503d}}|https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26138) Backport HBASE-25733 "Upgrade opentelemetry to 1.0.1" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26138.
--
Resolution: Fixed
> Backport HBASE-25733 "Upgrade opentelemetry to 1.0.1" to branch-2
> -
>
> Key: HBASE-26138
> URL: https://issues.apache.org/jira/browse/HBASE-26138
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 15/17 commits of HBASE-22120, original commit
> [{{b714889}}|https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26137) Backport HBASE-25732 "Change the command line argument for tracing after upgrading opentelemtry to 1.0.0" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26137.
--
Resolution: Fixed
commit pushed
https://github.com/apache/hbase/commit/d8f1ffa4d3731af654bf7ad94ee83dfa285fe168
> Backport HBASE-25732 "Change the command line argument for tracing after
> upgrading opentelemtry to 1.0.0" to branch-2
> -
>
> Key: HBASE-26137
> URL: https://issues.apache.org/jira/browse/HBASE-26137
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 14/17 commits of HBASE-22120, original commit
> [{{8df9beb}}|https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26136) Backport HBASE-25723 "Temporarily remove the trace support for RegionScanner.next" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26136.
--
Resolution: Fixed
commit pushed to
[https://github.com/apache/hbase/commit/817cfe1bf2c23de8cae178e0f96462ab5da33a59]
Thanks [~zhangduo] for reviewing.
> Backport HBASE-25723 "Temporarily remove the trace support for
> RegionScanner.next" to branch-2
> --
>
> Key: HBASE-26136
> URL: https://issues.apache.org/jira/browse/HBASE-26136
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 13/17 commits of HBASE-22120, original commit
> [{{7f90c22}}|https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26135) Backport HBASE-25616 "Upgrade opentelemetry to 1.0.0" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26135.
--
Resolution: Fixed
> Backport HBASE-25616 "Upgrade opentelemetry to 1.0.0" to branch-2
> -
>
> Key: HBASE-26135
> URL: https://issues.apache.org/jira/browse/HBASE-26135
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 12/17 commits of HBASE-22120, original commits
> [{{8399293}}|https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26134) Backport HBASE-25617 "Revisit the span names" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26134.
--
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Backport HBASE-25617 "Revisit the span names" to branch-2
> -
>
> Key: HBASE-26134
> URL: https://issues.apache.org/jira/browse/HBASE-26134
> Project: HBase
> Issue Type: Sub-task
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> _11/17 commits of HBASE-22120, original commit
> [{{8d68f8c}}|https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294]_
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26133) Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26133. -- Resolution: Fixed > Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2 > -- > > Key: HBASE-26133 > URL: https://issues.apache.org/jira/browse/HBASE-26133 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: trace, tracing > Fix For: 2.5.0 > > > 10/17 commits of HBASE-22120, original commit > f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26132) Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26132. -- Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Tags: trace Resolution: Fixed [https://github.com/apache/hbase/commit/9724df0e49245bd2bf46b126b76a8ab6f02f0f32] committed > Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to > branch-2 > --- > > Key: HBASE-26132 > URL: https://issues.apache.org/jira/browse/HBASE-26132 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26131) Backport HBASE-25484 "Add trace support for WAL sync" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26131. -- Resolution: Fixed > Backport HBASE-25484 "Add trace support for WAL sync" to branch-2 > - > > Key: HBASE-26131 > URL: https://issues.apache.org/jira/browse/HBASE-26131 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 8/17 commits of HBASE-22120, original commit > 2be2c63f0d3917a243b74af9754cbfc805b858d1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26130) Backport HBASE-25455 "Add trace support for HRegion read/write operation" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26130. -- Resolution: Fixed > Backport HBASE-25455 "Add trace support for HRegion read/write operation" to > branch-2 > - > > Key: HBASE-26130 > URL: https://issues.apache.org/jira/browse/HBASE-26130 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > 7/17 commits of HBASE-22120, original commit > 03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26129) Backport HBASE-25481 "Add host and port attribute when tracing rpc call at client side" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26129. -- Fix Version/s: 2.5.0 Resolution: Fixed > Backport HBASE-25481 "Add host and port attribute when tracing rpc call at > client side" to branch-2 > --- > > Key: HBASE-26129 > URL: https://issues.apache.org/jira/browse/HBASE-26129 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 6/17 commits of HBASE-22120, original commit > ae2c62ffaad5ba4c976b0a79c10a365edf2844fd -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26128) Backport HBASE-25454 "Add trace support for connection registry" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26128. -- Resolution: Fixed > Backport HBASE-25454 "Add trace support for connection registry" to branch-2 > > > Key: HBASE-26128 > URL: https://issues.apache.org/jira/browse/HBASE-26128 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: trace > Fix For: 2.5.0 > > > 5/17 commits of HBASE-22120, original commit > [dcb78bd4bda4a4ae13d863df8aec266031e5bc93|https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93] > and merged conflicts after rebasing on HBASE-26150 (branch-2 commit > [620806e3fb27947fa3b36f24921060e328f4b39d|https://github.com/apache/hbase/commit/620806e3fb27947fa3b36f24921060e328f4b39d]) > with commit > [63d4970de451bf234f2ddbda949995b1420e525b|https://github.com/apache/hbase/commit/63d4970de451bf234f2ddbda949995b1420e525b] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26127) Backport HBASE-23898 "Add trace support for simple apis in async client" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26127. -- Resolution: Fixed > Backport HBASE-23898 "Add trace support for simple apis in async client" to > branch-2 > > > Key: HBASE-26127 > URL: https://issues.apache.org/jira/browse/HBASE-26127 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 4/17 commits of HBASE-22120, original commit > https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26168) Backport HBASE-25811 for fixing nightly tests with error of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
Tak-Lon (Stephen) Wu created HBASE-26168:
Summary: Backport HBASE-25811 for fixing nightly tests with error
of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
Key: HBASE-26168
URL: https://issues.apache.org/jira/browse/HBASE-26168
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
see nightly error
[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/313/artifact/output-integration/hadoop-2/mr-importtsv.err/*view*/]
{code:java}
Caused by: java.lang.NoClassDefFoundError:
io/opentelemetry/api/GlobalOpenTelemetry
at org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:33)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:398)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:589)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:45761)
at
org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:50)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:372)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
... 32 more
Caused by: java.lang.ClassNotFoundException:
io.opentelemetry.api.GlobalOpenTelemetry
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 43 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26126) Backport HBASE-25424 "Find a way to config OpenTelemetry tracing without directly depending on opentelemetry-sdk" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26126. -- Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Release Note: https://github.com/apache/hbase/commit/3d29c0c2b4520edb06c0c5d3674cdb6547a57651 Resolution: Fixed https://github.com/apache/hbase/commit/3d29c0c2b4520edb06c0c5d3674cdb6547a57651 pushed to feature branch HBASE-25853 > Backport HBASE-25424 "Find a way to config OpenTelemetry tracing without > directly depending on opentelemetry-sdk" to branch-2 > - > > Key: HBASE-26126 > URL: https://issues.apache.org/jira/browse/HBASE-26126 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > [3/17 commits of > HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], original > commit of HBASE-25424 was > [https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d] -- This message was sent by Atlassian Jira (v8.3.4#803005)
