[jira] [Resolved] (HBASE-27759) Release 2.4.17
[ https://issues.apache.org/jira/browse/HBASE-27759?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-27759. -- Resolution: Fixed > Release 2.4.17 > -- > > Key: HBASE-27759 > URL: https://issues.apache.org/jira/browse/HBASE-27759 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28162) CLONE - Complete Release 2.4.17
Tak-Lon (Stephen) Wu created HBASE-28162: Summary: CLONE - Complete Release 2.4.17 Key: HBASE-28162 URL: https://issues.apache.org/jira/browse/HBASE-28162 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu # Release the artifacts on repository.apache.org # Move the binaries from dist-dev to dist-release # Add xml to download page # Push tag 2.4.17RC0 as tag rel/2.4.17 # Release 2.4.17 on JIRA https://issues.apache.org/jira/projects/HBASE/versions/12352760 # Add release data on [https://reporter.apache.org/addrelease.html?hbase] # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28164) CLONE - Add 2.4.17 to download page
Tak-Lon (Stephen) Wu created HBASE-28164: Summary: CLONE - Add 2.4.17 to download page Key: HBASE-28164 URL: https://issues.apache.org/jira/browse/HBASE-28164 Project: HBase Issue Type: Sub-task Components: website Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu Fix For: 3.0.0-alpha-4 need a PR to add rel/2.4.17 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28161) Release hbase-connectors 1.0.1
Tak-Lon (Stephen) Wu created HBASE-28161: Summary: Release hbase-connectors 1.0.1 Key: HBASE-28161 URL: https://issues.apache.org/jira/browse/HBASE-28161 Project: HBase Issue Type: Umbrella Components: community Reporter: Duo Zhang Assignee: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28163) CLONE - Put up 2.4.17RC0
Tak-Lon (Stephen) Wu created HBASE-28163: Summary: CLONE - Put up 2.4.17RC0 Key: HBASE-28163 URL: https://issues.apache.org/jira/browse/HBASE-28163 Project: HBase Issue Type: Sub-task Components: community Reporter: Tak-Lon (Stephen) Wu Assignee: Tak-Lon (Stephen) Wu send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28165) Fix hbase-connectors for release scripts
Tak-Lon (Stephen) Wu created HBASE-28165: Summary: Fix hbase-connectors for release scripts Key: HBASE-28165 URL: https://issues.apache.org/jira/browse/HBASE-28165 Project: HBase Issue Type: Bug Reporter: Tak-Lon (Stephen) Wu During the release for hbase-connectors 1.0.1 HBASE-28161, I found several issues 1. CHANGELOG.md was being used but our release script is using CHANGES.md to create new changes. 2. the do-release.sh script isn't working because of due 1/, I hacked it to make it work, but this would bring a backward compatible issues. Proposing to fix it after the releases, because no matter what the archive does not have the CHANGES.md for hbase-connectors. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28163) Put up hbase-connectors 1.0.1RC0
[ https://issues.apache.org/jira/browse/HBASE-28163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28163. -- Hadoop Flags: Reviewed Resolution: Fixed > Put up hbase-connectors 1.0.1RC0 > - > > Key: HBASE-28163 > URL: https://issues.apache.org/jira/browse/HBASE-28163 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > send out vote for 2.4.17RC0 -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28164) Add hbase-connectos 1.0.1 to download page
[ https://issues.apache.org/jira/browse/HBASE-28164?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28164. -- Resolution: Fixed > Add hbase-connectos 1.0.1 to download page > -- > > Key: HBASE-28164 > URL: https://issues.apache.org/jira/browse/HBASE-28164 > Project: HBase > Issue Type: Sub-task > Components: website >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 4.0.0-alpha-1 > > > need a PR to add hbase-connectors rel/1.0.1 to download page. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28162) Complete release hbase-connectors 1.0.1
[ https://issues.apache.org/jira/browse/HBASE-28162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28162. -- Release Note: added release note at https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12310753&version=12345436 Resolution: Fixed > Complete release hbase-connectors 1.0.1 > --- > > Key: HBASE-28162 > URL: https://issues.apache.org/jira/browse/HBASE-28162 > Project: HBase > Issue Type: Sub-task > Components: community >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > # Release the artifacts on repository.apache.org > # Move the binaries from dist-dev to dist-release > # Add xml to download page > # Push tag 1.0.1RC as tag rel/1.0.1 > # Release 1.0.1 on JIRA > https://issues.apache.org/jira/projects/HBASE/versions/12345436 > # Add release data on [https://reporter.apache.org/addrelease.html?hbase] > # Send announcement email -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28161) Release hbase-connectors 1.0.1
[ https://issues.apache.org/jira/browse/HBASE-28161?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-28161. -- Resolution: Fixed > Release hbase-connectors 1.0.1 > -- > > Key: HBASE-28161 > URL: https://issues.apache.org/jira/browse/HBASE-28161 > Project: HBase > Issue Type: Umbrella > Components: community >Reporter: Duo Zhang >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-20400) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
Tak Lon (Stephen) Wu created HBASE-20400: Summary: Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable Key: HBASE-20400 URL: https://issues.apache.org/jira/browse/HBASE-20400 Project: HBase Issue Type: Improvement Components: master, Performance Affects Versions: 2.0.0-beta-1, 3.0.0, 1.5.0, 1.4.4, 2.0.0 Reporter: Tak Lon (Stephen) Wu When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for notification (notify) from the fs.delete file thread. there might be two situation need to tune the MAX_WAIT in CleanerContext or make a configuration property for waitIfNotFinished when LogClearner call getResult. 1. fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long 2. getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20401) Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable
Tak Lon (Stephen) Wu created HBASE-20401: Summary: Make `MAX_WAIT` and `waitIfNotFinished` in CleanerContext configurable Key: HBASE-20401 URL: https://issues.apache.org/jira/browse/HBASE-20401 Project: HBase Issue Type: Improvement Affects Versions: 2.0.0-beta-1, 3.0.0, 1.5.0, 1.4.4, 2.0.0 Reporter: Tak Lon (Stephen) Wu When backporting HBASE-18309 in HBASE-20352, the deleteFiles calls CleanerContext.java#getResult with a waitIfNotFinished timeout to wait for notification (notify) from the fs.delete file thread. there might be two situation need to tune the MAX_WAIT in CleanerContext or waitIfNotFinished when LogClearner call getResult. # fs.delete never complete (strange but possible), then we need to wait for a max of 60 seconds. here, 60 seconds might be too long # getResult is waiting in the period of 500 milliseconds, but the fs.delete has completed and setFromClear is set but yet notify(). one might want to tune this 500 milliseconds to 200 or less . -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20555) Backport HBASE-18083 and related changes in branch-1
Tak Lon (Stephen) Wu created HBASE-20555: Summary: Backport HBASE-18083 and related changes in branch-1 Key: HBASE-20555 URL: https://issues.apache.org/jira/browse/HBASE-20555 Project: HBase Issue Type: Umbrella Components: HFile, snapshots Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu This will be the umbrella JIRA for backporting HBASE-18083 of `Make large/small file clean thread number configurable in HFileCleaner` from HBase's branch-2 to HBase's branch-1 that will needs a total of 4 sub-tasks that backport HBASE-16490, HBASE-17215, HBASE-17854 and then HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20556) Backport HBASE-16490 to branch-1
Tak Lon (Stephen) Wu created HBASE-20556: Summary: Backport HBASE-16490 to branch-1 Key: HBASE-20556 URL: https://issues.apache.org/jira/browse/HBASE-20556 Project: HBase Issue Type: Sub-task Components: HFile, snapshots Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu As part of HBASE-20555, HBASE-16490 is the first patch that is needed for backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20557) Backport HBASE-17215 to branch-1
Tak Lon (Stephen) Wu created HBASE-20557: Summary: Backport HBASE-17215 to branch-1 Key: HBASE-20557 URL: https://issues.apache.org/jira/browse/HBASE-20557 Project: HBase Issue Type: Sub-task Components: HFile, master Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu As part of HBASE-20555, HBASE-17215 is the second patch that is needed for backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20558) Backport HBASE-17854 to branch-1
Tak Lon (Stephen) Wu created HBASE-20558: Summary: Backport HBASE-17854 to branch-1 Key: HBASE-20558 URL: https://issues.apache.org/jira/browse/HBASE-20558 Project: HBase Issue Type: Sub-task Components: HFile Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu As part of HBASE-20555, HBASE-17854 is the third patch that is needed for backporting HBASE-18083 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20559) Backport HBASE-18083 to branch-1
Tak Lon (Stephen) Wu created HBASE-20559: Summary: Backport HBASE-18083 to branch-1 Key: HBASE-20559 URL: https://issues.apache.org/jira/browse/HBASE-20559 Project: HBase Issue Type: Sub-task Components: HFile Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu As part of HBASE-20555, HBASE-18083 --is the fourth/last patch that is needed for backporting.-- -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20608) Remove build option of error prone profile for branch-1 after HBASE-12350
Tak Lon (Stephen) Wu created HBASE-20608: Summary: Remove build option of error prone profile for branch-1 after HBASE-12350 Key: HBASE-20608 URL: https://issues.apache.org/jira/browse/HBASE-20608 Project: HBase Issue Type: Task Components: build Affects Versions: 1.4.4, 1.4.5 Reporter: Tak Lon (Stephen) Wu Assignee: Mike Drob After HBASE-12350, error prone profile was introduced to back to branch-1 and branch-2. however branch-1 is still building with JDK 7 and is incompatible with this error prone profile such that `mvn test-compile` failed since then. Open this issue to track the removal of `-PerrorProne` in the build command (in Jenkins) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-20837) Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml
Tak Lon (Stephen) Wu created HBASE-20837: Summary: Sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml Key: HBASE-20837 URL: https://issues.apache.org/jira/browse/HBASE-20837 Project: HBase Issue Type: Task Affects Versions: 1.4.5, 2.0.1, 3.0.0 Reporter: Tak Lon (Stephen) Wu While working on HBASE-20557 contribution, we figured out that the checkstyle build target (ImportOrder's `groups` [http://checkstyle.sourceforge.net/config_imports.html] ) was different from the development supported IDE formatter, we would provide a fix here to sync between dev-support/hbase_eclipse_formatter.xml and hbase/checkstyle.xml This might need to backport the changes of master to branch-1 and branch-2 as well. Before this change, this is what checkstyle is expecting ``` import com.google.common.annotations.VisibleForTesting; import java.io.IOException; import java.util.ArrayList; import java.util.List; import java.util.Map; import org.apache.commons.logging.Log; import org.apache.commons.logging.LogFactory; import org.apache.hadoop.conf.Configuration; import org.apache.hadoop.hbase.classification.InterfaceAudience; import org.apache.hadoop.hbase.conf.ConfigurationObserver; ``` -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (HBASE-20555) Backport HBASE-18083 and related changes in branch-1
[ https://issues.apache.org/jira/browse/HBASE-20555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak Lon (Stephen) Wu resolved HBASE-20555. -- Resolution: Fixed Fix Version/s: 1.4.6 1.5.0 Thanks [~zyork] [~apurtell] [~carp84] for the comments, and reviewing the subtasks. since all ports have been resolve, closing this umbrella issue > Backport HBASE-18083 and related changes in branch-1 > > > Key: HBASE-20555 > URL: https://issues.apache.org/jira/browse/HBASE-20555 > Project: HBase > Issue Type: Umbrella > Components: HFile, snapshots >Affects Versions: 1.4.4, 1.4.5 >Reporter: Tak Lon (Stephen) Wu >Assignee: Tak Lon (Stephen) Wu >Priority: Major > Fix For: 1.5.0, 1.4.6 > > > This will be the umbrella JIRA for backporting HBASE-18083 of `Make > large/small file clean thread number configurable in HFileCleaner` from > HBase's branch-2 to HBase's branch-1 that will needs a total of 4 sub-tasks > that backport HBASE-16490, HBASE-17215, HBASE-17854 and then HBASE-18083 > The goal is to improve HFile cleaning performance that has been introduced in > branch-2 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately
Tak Lon (Stephen) Wu created HBASE-21011:
Summary: Provide CLI option to run oldwals and hfiles cleaner
separately
Key: HBASE-21011
URL: https://issues.apache.org/jira/browse/HBASE-21011
Project: HBase
Issue Type: Improvement
Components: Admin, Client
Affects Versions: 1.4.6, 3.0.0, 2.1.1
Reporter: Tak Lon (Stephen) Wu
Assignee: Tak Lon (Stephen) Wu
Existing logic of cleaner chore is first execute HFiles cleaner and then
oldwals cleaner in a request, and only return succeed if both completes.
There is a use case of running only oldwals cleaner because oldwals uses all
the disk space, and running HFiles cleaner is too slow because either the
amount of old HFiles or directories
are too much. So, this change provide the flexibility for those disabled
cleaner by default or would like to execute admin command to run oldwals and
HFiles cleaning procedure individually.
NOTE that we keep the default as running both of them for backward
compatibility, e.g. the proposed admin CLI options are
{{ hbase> cleaner_chore_run}}
{{ hbase> cleaner_chore_run 'hfiles'}}
{{ hbase> cleaner_chore_run 'oldwals'}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-21011) Provide CLI option to run oldwals and hfiles cleaner separately when cleaner chore is disabled
[
https://issues.apache.org/jira/browse/HBASE-21011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu resolved HBASE-21011.
--
Resolution: Won't Fix
Aligned with Reid's comment, operator has few options to get around with this
situation, e.g. turn on cleaner chore. close it and marked as won't fix.
> Provide CLI option to run oldwals and hfiles cleaner separately when cleaner
> chore is disabled
> --
>
> Key: HBASE-21011
> URL: https://issues.apache.org/jira/browse/HBASE-21011
> Project: HBase
> Issue Type: Improvement
> Components: Admin, Client
>Affects Versions: 3.0.0, 1.4.6, 2.1.1
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Attachments: HBASE-21011.master.001.patch,
> HBASE-21011.master.002.patch, HBASE-21011.master.003.patch,
> HBASE-21011.master.004.patch
>
>
> There is a corner case when cleaner chore for HFiles and oldwals is disabled,
> admin/user needs to manually execute admin command {{cleaner_chore_run}} to
> clean the old HFiles and oldwals. Existing logic of {{cleaner_chore_run}} is
> to [firstly trigger the HFiles cleaner and then oldwals
> cleaner|https://github.com/taklwu/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/master/MasterRpcServices.java#L1414-L1420],
> and only return succeed if both completes.
> but when running this {{cleaner_chore_run}} command, there is a potential use
> case that admin would like trigger the cleaner for only oldwals or hfiles but
> still keep the automatic cleaner chore disabled. So, this change aims to
> provide support for this corner case, and provide flexibility for those user
> with cleaner chore disabled by default to execute admin CLI to run oldwals
> and HFiles cleaning procedure individually.
> NOTE that {{cleaner_chore_run}} was introduced in HBASE-17280, this patch
> added options 'hfiles' and 'oldwals' to it. Also fix default behavior of
> {{cleaner_chore_run}} will be only ran when cleaner chore is set to disabled,
> e.g. the proposed admin CLI options are
> {noformat}
> hbase> cleaner_chore_run # this was introduced in HBASE-17280,
> but changed the behavior to only ran when cleaner chore is set to disabled
> hbase> cleaner_chore_run 'hfiles' # added, ran when cleaner chore is set
> to disabled
> hbase> cleaner_chore_run 'oldwals' # added, ran when cleaner chore is set
> to disabled
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21181) Use the same filesystem for wal archive directory and wal directory
Tak Lon (Stephen) Wu created HBASE-21181:
Summary: Use the same filesystem for wal archive directory and wal
directory
Key: HBASE-21181
URL: https://issues.apache.org/jira/browse/HBASE-21181
Project: HBase
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.1
Reporter: Tak Lon (Stephen) Wu
Assignee: Tak Lon (Stephen) Wu
when {{hbase.wal.dir}} is set to any filesystem other than the same filesystem
used by rootDir e.g. {{hbase.wal.dir}} set to {{hdfs://something/wal}} and
{{hbase.rootdir}} set to {{s3://something}}, before this change, WAL archive
directory ({{walArchiveDir}}) cannot be created and failed the
WALProcedureStore on HMaster.
The issue is that WAL archive directory was considered to be collocated with
{{hbase.rootdir}} and creates a subdirectory under it. this logic needs to be
updated.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (HBASE-20837) Make IDE configuration for import order match that in our checkstyle module
[
https://issues.apache.org/jira/browse/HBASE-20837?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak Lon (Stephen) Wu resolved HBASE-20837.
--
Resolution: Won't Fix
> Make IDE configuration for import order match that in our checkstyle module
> ---
>
> Key: HBASE-20837
> URL: https://issues.apache.org/jira/browse/HBASE-20837
> Project: HBase
> Issue Type: Improvement
> Components: community
>Affects Versions: 3.0.0, 2.0.1, 1.4.5
>Reporter: Tak Lon (Stephen) Wu
>Assignee: Tak Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0, 1.5.0, 2.2.0
>
> Attachments: HBASE-20837.branch-1.001.patch,
> HBASE-20837.branch-1.002.patch, HBASE-20837.branch-1.003.patch,
> HBASE-20837.branch-2.001.patch, HBASE-20837.branch-2.002.patch,
> HBASE-20837.branch-2.003.patch, HBASE-20837.master.001.patch,
> HBASE-20837.master.002.patch, HBASE-20837.master.003.patch, IDEA import
> layout.png, hbase-intellij-formatter.xml
>
>
> While working on HBASE-20557 contribution, we figured out that the checkstyle
> build target (ImportOrder's `groups`
> [http://checkstyle.sourceforge.net/config_imports.html] ) was different from
> the development supported IDE (e.g. IntelliJ and Eclipse) formatter, we would
> provide a fix here to sync between
> [dev-support/hbase_eclipse_formatter.xml|https://github.com/apache/hbase/blob/master/dev-support/hbase_eclipse_formatter.xml]
> and
> [hbase/checkstyle.xml|https://github.com/apache/hbase/blob/master/hbase-checkstyle/src/main/resources/hbase/checkstyle.xml]
> This might need to backport the changes of master to branch-1 and branch-2 as
> well.
> Before this change, this is what checkstyle is expecting for import order
>
> {code:java}
> import com.google.common.annotations.VisibleForTesting;
> import java.io.IOException;
> import java.util.ArrayList;
> import java.util.List;
> import java.util.Map;
> import org.apache.commons.logging.Log;
> import org.apache.commons.logging.LogFactory;
> import org.apache.hadoop.conf.Configuration;
> import org.apache.hadoop.hbase.classification.InterfaceAudience;
> import org.apache.hadoop.hbase.conf.ConfigurationObserver;{code}
>
> And the proposed import order with the respect to HBASE-19262 and HBASE-19552
> should be
>
> !IDEA import layout.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-21318) Make RefreshHFilesClient runnable
Tak Lon (Stephen) Wu created HBASE-21318: Summary: Make RefreshHFilesClient runnable Key: HBASE-21318 URL: https://issues.apache.org/jira/browse/HBASE-21318 Project: HBase Issue Type: Improvement Components: HFile Affects Versions: 3.0.0, 1.5.0, 2.1.2 Reporter: Tak Lon (Stephen) Wu Assignee: Tak Lon (Stephen) Wu Other than when user enables hbase.coprocessor.region.classes with RefreshHFilesEndPoint, user can also run this client as tool runner class/CLI and calls refresh HFiles directly. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (HBASE-21963) Add a script for building and verifying release candidate
Tak Lon (Stephen) Wu created HBASE-21963:
Summary: Add a script for building and verifying release candidate
Key: HBASE-21963
URL: https://issues.apache.org/jira/browse/HBASE-21963
Project: HBase
Issue Type: Test
Components: release
Affects Versions: 2.1.3, 3.0.0
Reporter: Tak Lon (Stephen) Wu
Assignee: Tak Lon (Stephen) Wu
During the release vote for HBase 2.1.3RC1, a driver/helper script was
mentioned and can potentially help contributors prepare to vote for a release
candidate. As recommended, we decided to move toward this tool to under
{{dev-support/}}
Here the driver script provides the following automation:
1. Import and check publisher key(s)
2. Checksum of sources and binaries
3. Signature of sources and binaries
4. Rat check
5. Built from source
6. Verify unit tests
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (HBASE-24749) Direct insert HFiles and Persist in-memory HFile tracking
Tak-Lon (Stephen) Wu created HBASE-24749:
Summary: Direct insert HFiles and Persist in-memory HFile tracking
Key: HBASE-24749
URL: https://issues.apache.org/jira/browse/HBASE-24749
Project: HBase
Issue Type: Umbrella
Components: Compaction, HFile, Zookeeper
Affects Versions: 3.0.0-alpha-1
Reporter: Tak-Lon (Stephen) Wu
Attachments: 1B100m-25m25m-performance.pdf, Apache HBase - Direct
insert HFiles and Persist in-memory HFile tracking.pdf
We propose removing the {{.tmp}} directory used in the commit stage for common
HFile operations such as flush and compaction to improve the write throughput
and latency on object stores. Specifically for S3 filesystems, this will also
mitigate read-after-write inconsistencies caused by immediate HFiles validation
after moving the HFile(s) to data directory.
Please see attached for this proposal and the initial result captured with 25m
(25m operations) and 1B (100m operations) YCSB workload A LOAD and RUN, and
workload C RUN result.
the goal of this JIRA is to discuss with the community if the proposed
improvement on the object stores use case makes senses and if we miss anything
should be included.
Improvement Highlights
1. Lower write latency, especially the p99+
2. Higher write throughput on flush and compaction
3. Lower MTTR on region (re)open or assignment
4. Remove consistent check dependencies (e.g. DynamoDB) supported by file
system imple
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-24833) Bootstrap should not delete the META table directory if it's not partial
Tak-Lon (Stephen) Wu created HBASE-24833:
Summary: Bootstrap should not delete the META table directory if
it's not partial
Key: HBASE-24833
URL: https://issues.apache.org/jira/browse/HBASE-24833
Project: HBase
Issue Type: Umbrella
Reporter: Tak-Lon (Stephen) Wu
this issues were discussed in
[PR#2113|https://github.com/apache/hbase/pull/2113] as part of HBASE-24286, and
it is a dependencies before we solve HBASE-24286.
The changes were introduced in [HBASE-24471
|https://github.com/apache/hbase/commit/4d5efec76718032a1e55024fd5133409e4be3cb8#diff-21659161b1393e6632730dcbea205fd8R70-R89]
that partial meta was introduced and `partial` was defined as
InitMetaProcedure did not succeed and INIT_META_ASSIGN_META was not completed.
{{ private static void writeFsLayout(Path rootDir, Configuration conf) throws
IOException {
LOG.info("BOOTSTRAP: creating hbase:meta region");
FileSystem fs = rootDir.getFileSystem(conf);
Path tableDir = CommonFSUtils.getTableDir(rootDir,
TableName.META_TABLE_NAME);
if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
LOG.warn("Can not delete partial created meta table, continue...");
} }}
however, in the cloud use case where HFiles store on S3, WALs store on HDFS, ZK
data are stored within the cluster. Here, Zk data and WALs cannot be retained
(HDFS was associated with cloud instance and was terminated together) when
cluster recreates on the flushed HFiles, and existing meta are always
considered as partial and deleted in `INIT_META_WRITE_FS_LAYOUT` during
bootstrap. as a result, the recreate cluster starts with a empty meta table,
either the cluster hangs during the master initialization (branch-2) because
table states of namespace table cannot be assigned, or starts as a fresh
cluster without any region assigned and table opens (may need HBCK to rebuild
the meta).
Potential solution suggested by Anoop
bq. In case of HM start and the bootstrap we create the ClusterID and write to
FS and then to zk and then create the META table FS layout. So in a cluster
recreate, we will see clusterID is there in FS and also the META FS layout but
no clusterID in zk. Ya seems we can use this as indication for cluster recreate
over existing data. In HM start, this is some thing we need to check at 1st
itself and track. If this mode is true, later when (if) we do
INIT_META_WRITE_FS_LAYOUT , we should not delete the META dir. As part of the
Bootstrap when we write that proc to MasterProcWal, we can include this mode
(boolean) info also. This is a protobuf message anyways. So even if this HM got
killed and restarted (at a point where the clusterId was written to zk but the
Meta FS layout part was not reached) we can use the info added as part of the
bootstrap wal entry and make sure NOT to delete the meta dir.
In this JIRA, we're going to fix the `partial` definition when we found cluster
ID was stored in HFiles but ZK were deleted or fresh on cluster creates.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25391) Flush directly into data directory, skip rename when committing flush
Tak-Lon (Stephen) Wu created HBASE-25391:
Summary: Flush directly into data directory, skip rename when
committing flush
Key: HBASE-25391
URL: https://issues.apache.org/jira/browse/HBASE-25391
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}When flushing memstore snapshot to HFile, we write it directly
to the data directory.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25392) Direct insert compacted HFiles into data directory.
Tak-Lon (Stephen) Wu created HBASE-25392:
Summary: Direct insert compacted HFiles into data directory.
Key: HBASE-25392
URL: https://issues.apache.org/jira/browse/HBASE-25392
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}when performing compaction, write directory to the data
directory {color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25393) Support split and merge region with direct insert into CF directory
Tak-Lon (Stephen) Wu created HBASE-25393:
Summary: Support split and merge region with direct insert into CF
directory
Key: HBASE-25393
URL: https://issues.apache.org/jira/browse/HBASE-25393
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}Support region SPLIT and MERGE with direct insert HFiles into
column family directory{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25394) Support Snapshot related operation with direct insert HFiles into data/CF directory
Tak-Lon (Stephen) Wu created HBASE-25394:
Summary: Support Snapshot related operation with direct insert
HFiles into data/CF directory
Key: HBASE-25394
URL: https://issues.apache.org/jira/browse/HBASE-25394
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}Support restore snapshot, clone snapshot with direct insert into
data directory{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
Tak-Lon (Stephen) Wu created HBASE-25395: Summary: Introduce PersistedStoreEngine and PersistedStoreFileManager Key: HBASE-25395 URL: https://issues.apache.org/jira/browse/HBASE-25395 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Create a new wrapper store engine on top of the default store engine, as well as, introducing a new store file manager that keep tracking path of HFiles of user tables into a hbase self-managed system table (further will have a different tasks to merge into meta table and check throughput do not change much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25396) Integtration with Direct insert into data/cf directory
Tak-Lon (Stephen) Wu created HBASE-25396: Summary: Integtration with Direct insert into data/cf directory Key: HBASE-25396 URL: https://issues.apache.org/jira/browse/HBASE-25396 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Utilizes direct insert compactor, file writer, committer and etc while constructing the PersistedStoreEngine -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25397) Migrate hbase:storefile system table into editable meta
Tak-Lon (Stephen) Wu created HBASE-25397:
Summary: Migrate hbase:storefile system table into editable meta
Key: HBASE-25397
URL: https://issues.apache.org/jira/browse/HBASE-25397
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}editable meta supported by HBASE-23055 allow us to write new
column family to hbase:meta, such that we will introduce a new column family
cf:storefile to hbase:meta with different trackable columns to served as the
in-memory tracking of e.g. `storefiles` and `compactedfile` in
StoreFileManager.{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25398) Cleanup unused/unneeded temporarily HFiles
Tak-Lon (Stephen) Wu created HBASE-25398: Summary: Cleanup unused/unneeded temporarily HFiles Key: HBASE-25398 URL: https://issues.apache.org/jira/browse/HBASE-25398 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu Cleanup records and write a new cleaner to move unneeded files from data directory into archives (other than compaction) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25399) Write HBCK tools to rebuild the tracking
Tak-Lon (Stephen) Wu created HBASE-25399:
Summary: Write HBCK tools to rebuild the tracking
Key: HBASE-25399
URL: https://issues.apache.org/jira/browse/HBASE-25399
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
{color:#00}support online region repairing, and maybe offline repairing as
well
{color}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25395) Introduce PersistedStoreEngine and PersistedStoreFileManager
[ https://issues.apache.org/jira/browse/HBASE-25395?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-25395. -- Release Note: Added `hbase.storefile.tracking.persist.enabled` to enable `PersistedStoreEngine` and `PersistedStoreFileManager` that fork the file paths of in-memory object `storefile` in StoreFileManager into a system table `hbase:storefile`. This persisted paths will be reused when region (re)opens on existing store without scanning the file storage. Resolution: Fixed > Introduce PersistedStoreEngine and PersistedStoreFileManager > - > > Key: HBASE-25395 > URL: https://issues.apache.org/jira/browse/HBASE-25395 > Project: HBase > Issue Type: Sub-task >Affects Versions: 3.0.0-alpha-1 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > Create a new wrapper store engine on top of the default store engine, as well > as, introducing a new store file manager that keep tracking path of HFiles of > user tables into a hbase self-managed system table (further will have a > different tasks to merge into meta table and check throughput do not change > much) -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-25921) Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
Tak-Lon (Stephen) Wu created HBASE-25921:
Summary: Fix Wrong FileSystem when running `filesystem` on
non-HDFS storage
Key: HBASE-25921
URL: https://issues.apache.org/jira/browse/HBASE-25921
Project: HBase
Issue Type: Bug
Components: hbase-operator-tools
Affects Versions: 1.2.0, 1.1.0, 1.0.0
Reporter: Tak-Lon (Stephen) Wu
Assignee: Tak-Lon (Stephen) Wu
recently we found a bug that when running {{hbck filesystem -f}} on a non-HDFS
and caused {{Wrong FS}} error and cannot move forward.
this change fix this problem and use {{FSUtils.getCurrentFileSystem}} to
respect the root directory of what we set with {{hbase.rootdir}} in stead of
just using {{fs.defaultFS}}
e.g. this is one of the captured ERROR message
{code}
Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
s3://xyz/data/default/abc, expected: hdfs://xyz/
at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:233)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1045)
at
org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1112)
at
org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1109)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1119)
at
org.apache.hadoop.hbase.util.FSUtils.listStatusWithStatusFilter(FSUtils.java:1530)
at
org.apache.hbase.hbck1.HFileCorruptionChecker.checkTableDir(HFileCorruptionChecker.java:320)
at
org.apache.hbase.hbck1.HFileCorruptionChecker.checkTables(HFileCorruptionChecker.java:431)
at org.apache.hbase.FileSystemFsck.fsck(FileSystemFsck.java:87)
at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:721)
at org.apache.hbase.HBCK2.run(HBCK2.java:631)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
at org.apache.hbase.HBCK2.main(HBCK2.java:865)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25921) Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
[
https://issues.apache.org/jira/browse/HBASE-25921?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-25921.
--
Fix Version/s: 1.2.0
Resolution: Fixed
> Fix Wrong FileSystem when running `filesystem` on non-HDFS storage
> --
>
> Key: HBASE-25921
> URL: https://issues.apache.org/jira/browse/HBASE-25921
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: 1.0.0, 1.1.0, 1.2.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Minor
> Fix For: 1.2.0
>
>
> recently we found a bug that when running {{hbck filesystem -f}} on a
> non-HDFS and caused {{Wrong FS}} error and cannot move forward.
> this change fix this problem and use {{FSUtils.getCurrentFileSystem}} to
> respect the root directory of what we set with {{hbase.rootdir}} in stead of
> just using {{fs.defaultFS}}
> e.g. this is one of the captured ERROR message
> {code}
> Exception in thread "main" java.lang.IllegalArgumentException: Wrong FS:
> s3://xyz/data/default/abc, expected: hdfs://xyz/
> at org.apache.hadoop.fs.FileSystem.checkPath(FileSystem.java:737)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.getPathName(DistributedFileSystem.java:233)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:1045)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.access$1000(DistributedFileSystem.java:131)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1112)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem$24.doCall(DistributedFileSystem.java:1109)
> at
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> at
> org.apache.hadoop.hdfs.DistributedFileSystem.listStatus(DistributedFileSystem.java:1119)
> at
> org.apache.hadoop.hbase.util.FSUtils.listStatusWithStatusFilter(FSUtils.java:1530)
> at
> org.apache.hbase.hbck1.HFileCorruptionChecker.checkTableDir(HFileCorruptionChecker.java:320)
> at
> org.apache.hbase.hbck1.HFileCorruptionChecker.checkTables(HFileCorruptionChecker.java:431)
> at org.apache.hbase.FileSystemFsck.fsck(FileSystemFsck.java:87)
> at org.apache.hbase.HBCK2.doCommandLine(HBCK2.java:721)
> at org.apache.hbase.HBCK2.run(HBCK2.java:631)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hbase.HBCK2.main(HBCK2.java:865)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-25965) Move delete of WAL directory out of LOG.info and avoid left over directory
Tak-Lon (Stephen) Wu created HBASE-25965:
Summary: Move delete of WAL directory out of LOG.info and avoid
left over directory
Key: HBASE-25965
URL: https://issues.apache.org/jira/browse/HBASE-25965
Project: HBase
Issue Type: Bug
Components: hbase-operator-tools
Affects Versions: 1.1.0
Reporter: Tak-Lon (Stephen) Wu
Followup the comment by [~anoopsamjohn] on HBASE-25921
{quote} Not on this patch. But here is an issue. The actual delete() call
happening within a Log statement. If INFO level not enabled, the delete wont
get called. Can we fix that also here pls? {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-25965) Move delete of WAL directory out of LOG.info and avoid left over directory
[
https://issues.apache.org/jira/browse/HBASE-25965?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-25965.
--
Fix Version/s: 1.2.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Move delete of WAL directory out of LOG.info and avoid left over directory
> --
>
> Key: HBASE-25965
> URL: https://issues.apache.org/jira/browse/HBASE-25965
> Project: HBase
> Issue Type: Bug
> Components: hbase-operator-tools
>Affects Versions: 1.1.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Minor
> Fix For: 1.2.0
>
>
> Followup the comment by [~anoopsamjohn] on HBASE-25921 (in
> [PR#88|https://github.com/apache/hbase-operator-tools/pull/88])
> {quote} Not on this patch. But here is an issue. The actual delete() call
> happening within a Log statement. If INFO level not enabled, the delete wont
> get called. Can we fix that also here pls? {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26124) Backport HBASE-25373 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26124: Summary: Backport HBASE-25373 to branch-2 Key: HBASE-26124 URL: https://issues.apache.org/jira/browse/HBASE-26124 Project: HBase Issue Type: Sub-task Components: tracing Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu [1/17 commits of HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], backporting HBASE-25373 Remove HTrace completely in code base and try to make use of OpenTelemetry We will need to remove some unimplemented tracing in AsyncRequestFutureImpl#getNewMultiActionRunnable that master branch does not have it and also upstream in master does not implement TraceRunnable with OpenTelemetry plugin -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26126) Backport HBASE-25424 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26126: Summary: Backport HBASE-25424 to branch-2 Key: HBASE-26126 URL: https://issues.apache.org/jira/browse/HBASE-26126 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26125) Backport HBASE-25401 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26125: Summary: Backport HBASE-25401 to branch-2 Key: HBASE-26125 URL: https://issues.apache.org/jira/browse/HBASE-26125 Project: HBase Issue Type: Sub-task Components: tracing Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26128) Backport HBASE-25454 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26128: Summary: Backport HBASE-25454 to branch-2 Key: HBASE-26128 URL: https://issues.apache.org/jira/browse/HBASE-26128 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26129) Backport HBASE-25481 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26129: Summary: Backport HBASE-25481 to branch-2 Key: HBASE-26129 URL: https://issues.apache.org/jira/browse/HBASE-26129 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26130) Backport HBASE-25455 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26130: Summary: Backport HBASE-25455 to branch-2 Key: HBASE-26130 URL: https://issues.apache.org/jira/browse/HBASE-26130 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26127) Backport HBASE-23898 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26127: Summary: Backport HBASE-23898 to branch-2 Key: HBASE-26127 URL: https://issues.apache.org/jira/browse/HBASE-26127 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26131) Backport HBASE-25484 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26131: Summary: Backport HBASE-25484 to branch-2 Key: HBASE-26131 URL: https://issues.apache.org/jira/browse/HBASE-26131 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26133) Backport HBASE-25591 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26133: Summary: Backport HBASE-25591 to branch-2 Key: HBASE-26133 URL: https://issues.apache.org/jira/browse/HBASE-26133 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26134) Backport HBASE-25617 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26134: Summary: Backport HBASE-25617 to branch-2 Key: HBASE-26134 URL: https://issues.apache.org/jira/browse/HBASE-26134 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26136) Backport HBASE-25723 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26136: Summary: Backport HBASE-25723 to branch-2 Key: HBASE-26136 URL: https://issues.apache.org/jira/browse/HBASE-26136 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26132) Backport HBASE-25535 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26132: Summary: Backport HBASE-25535 to branch-2 Key: HBASE-26132 URL: https://issues.apache.org/jira/browse/HBASE-26132 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26138) Backport HBASE-25733 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26138: Summary: Backport HBASE-25733 to branch-2 Key: HBASE-26138 URL: https://issues.apache.org/jira/browse/HBASE-26138 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26137) Backport HBASE-25732 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26137: Summary: Backport HBASE-25732 to branch-2 Key: HBASE-26137 URL: https://issues.apache.org/jira/browse/HBASE-26137 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26139) Backport HBASE-23762 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26139: Summary: Backport HBASE-23762 to branch-2 Key: HBASE-26139 URL: https://issues.apache.org/jira/browse/HBASE-26139 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26140) Backport HBASE-25778 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26140: Summary: Backport HBASE-25778 to branch-2 Key: HBASE-26140 URL: https://issues.apache.org/jira/browse/HBASE-26140 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26135) Backport HBASE-25616 to branch-2
Tak-Lon (Stephen) Wu created HBASE-26135: Summary: Backport HBASE-25616 to branch-2 Key: HBASE-26135 URL: https://issues.apache.org/jira/browse/HBASE-26135 Project: HBase Issue Type: Sub-task Reporter: Tak-Lon (Stephen) Wu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26141) Add tracing support for sync client on branch-2
Tak-Lon (Stephen) Wu created HBASE-26141: Summary: Add tracing support for sync client on branch-2 Key: HBASE-26141 URL: https://issues.apache.org/jira/browse/HBASE-26141 Project: HBase Issue Type: New Feature Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu Split from HBASE-25853, the original change of HBASE-22120 does not include tracing support for sync client, if we decide to support it in branch-2, we will need to come up our own solution. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26124) Backport HBASE-25373 "Remove HTrace completely in code base and try to make use of OpenTelemetry" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26124?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26124. -- Hadoop Flags: Reviewed Release Note: https://github.com/apache/hbase/commit/f0493016062267fc37e14659d9183673d42a8f1d Resolution: Fixed > Backport HBASE-25373 "Remove HTrace completely in code base and try to make > use of OpenTelemetry" to branch-2 > - > > Key: HBASE-26124 > URL: https://issues.apache.org/jira/browse/HBASE-26124 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0, 2.3.5, 2.4.5 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > [1/17 commits of > HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], backporting > HBASE-25373 Remove HTrace completely in code base and try to make use of > OpenTelemetry > We will need to remove some unimplemented tracing in > AsyncRequestFutureImpl#getNewMultiActionRunnable that master branch does not > have it and also upstream in master does not implement TraceRunnable with > OpenTelemetry plugin > the original commit on master is > https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26125) Backport HBASE-25401 "Add trace support for async call in rpc client" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26125?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26125. -- Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Release Note: https://github.com/apache/hbase/commit/ca096437d7e096b514ddda53ec2f97b85d90752d Resolution: Fixed > Backport HBASE-25401 "Add trace support for async call in rpc client" to > branch-2 > - > > Key: HBASE-26125 > URL: https://issues.apache.org/jira/browse/HBASE-26125 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > [2/17 commits of > HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], original > commit of HBASE-25401 was > [https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26126) Backport HBASE-25424 "Find a way to config OpenTelemetry tracing without directly depending on opentelemetry-sdk" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26126. -- Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Release Note: https://github.com/apache/hbase/commit/3d29c0c2b4520edb06c0c5d3674cdb6547a57651 Resolution: Fixed https://github.com/apache/hbase/commit/3d29c0c2b4520edb06c0c5d3674cdb6547a57651 pushed to feature branch HBASE-25853 > Backport HBASE-25424 "Find a way to config OpenTelemetry tracing without > directly depending on opentelemetry-sdk" to branch-2 > - > > Key: HBASE-26126 > URL: https://issues.apache.org/jira/browse/HBASE-26126 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > [3/17 commits of > HBASE-22120|https://github.com/apache/hbase/pull/2901/commits], original > commit of HBASE-25424 was > [https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26168) Backport HBASE-25811 for fixing nightly tests with error of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
Tak-Lon (Stephen) Wu created HBASE-26168:
Summary: Backport HBASE-25811 for fixing nightly tests with error
of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
Key: HBASE-26168
URL: https://issues.apache.org/jira/browse/HBASE-26168
Project: HBase
Issue Type: Sub-task
Reporter: Tak-Lon (Stephen) Wu
see nightly error
[https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/313/artifact/output-integration/hadoop-2/mr-importtsv.err/*view*/]
{code:java}
Caused by: java.lang.NoClassDefFoundError:
io/opentelemetry/api/GlobalOpenTelemetry
at org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:33)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:398)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
at
org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:589)
at
org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:45761)
at
org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:50)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:372)
at
org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
at
org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
... 32 more
Caused by: java.lang.ClassNotFoundException:
io.opentelemetry.api.GlobalOpenTelemetry
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
... 43 more
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26127) Backport HBASE-23898 "Add trace support for simple apis in async client" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26127?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26127. -- Resolution: Fixed > Backport HBASE-23898 "Add trace support for simple apis in async client" to > branch-2 > > > Key: HBASE-26127 > URL: https://issues.apache.org/jira/browse/HBASE-26127 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 4/17 commits of HBASE-22120, original commit > https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26128) Backport HBASE-25454 "Add trace support for connection registry" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26128?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26128. -- Resolution: Fixed > Backport HBASE-25454 "Add trace support for connection registry" to branch-2 > > > Key: HBASE-26128 > URL: https://issues.apache.org/jira/browse/HBASE-26128 > Project: HBase > Issue Type: Sub-task > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: trace > Fix For: 2.5.0 > > > 5/17 commits of HBASE-22120, original commit > [dcb78bd4bda4a4ae13d863df8aec266031e5bc93|https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93] > and merged conflicts after rebasing on HBASE-26150 (branch-2 commit > [620806e3fb27947fa3b36f24921060e328f4b39d|https://github.com/apache/hbase/commit/620806e3fb27947fa3b36f24921060e328f4b39d]) > with commit > [63d4970de451bf234f2ddbda949995b1420e525b|https://github.com/apache/hbase/commit/63d4970de451bf234f2ddbda949995b1420e525b] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26129) Backport HBASE-25481 "Add host and port attribute when tracing rpc call at client side" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26129?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26129. -- Fix Version/s: 2.5.0 Resolution: Fixed > Backport HBASE-25481 "Add host and port attribute when tracing rpc call at > client side" to branch-2 > --- > > Key: HBASE-26129 > URL: https://issues.apache.org/jira/browse/HBASE-26129 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 6/17 commits of HBASE-22120, original commit > ae2c62ffaad5ba4c976b0a79c10a365edf2844fd -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26130) Backport HBASE-25455 "Add trace support for HRegion read/write operation" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26130. -- Resolution: Fixed > Backport HBASE-25455 "Add trace support for HRegion read/write operation" to > branch-2 > - > > Key: HBASE-26130 > URL: https://issues.apache.org/jira/browse/HBASE-26130 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > > 7/17 commits of HBASE-22120, original commit > 03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26131) Backport HBASE-25484 "Add trace support for WAL sync" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26131. -- Resolution: Fixed > Backport HBASE-25484 "Add trace support for WAL sync" to branch-2 > - > > Key: HBASE-26131 > URL: https://issues.apache.org/jira/browse/HBASE-26131 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > 8/17 commits of HBASE-22120, original commit > 2be2c63f0d3917a243b74af9754cbfc805b858d1 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26132) Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26132?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26132. -- Fix Version/s: 2.5.0 Hadoop Flags: Reviewed Tags: trace Resolution: Fixed [https://github.com/apache/hbase/commit/9724df0e49245bd2bf46b126b76a8ab6f02f0f32] committed > Backport HBASE-25535 "Set span kind to CLIENT in AbstractRpcClient" to > branch-2 > --- > > Key: HBASE-26132 > URL: https://issues.apache.org/jira/browse/HBASE-26132 > Project: HBase > Issue Type: Sub-task >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26133) Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2
[ https://issues.apache.org/jira/browse/HBASE-26133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26133. -- Resolution: Fixed > Backport HBASE-25591 "Upgrade opentelemetry to 0.17.1" to branch-2 > -- > > Key: HBASE-26133 > URL: https://issues.apache.org/jira/browse/HBASE-26133 > Project: HBase > Issue Type: Sub-task >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Labels: trace, tracing > Fix For: 2.5.0 > > > 10/17 commits of HBASE-22120, original commit > f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-26134) Backport HBASE-25617 "Revisit the span names" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26134.
--
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Backport HBASE-25617 "Revisit the span names" to branch-2
> -
>
> Key: HBASE-26134
> URL: https://issues.apache.org/jira/browse/HBASE-26134
> Project: HBase
> Issue Type: Sub-task
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> _11/17 commits of HBASE-22120, original commit
> [{{8d68f8c}}|https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294]_
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26135) Backport HBASE-25616 "Upgrade opentelemetry to 1.0.0" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26135?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26135.
--
Resolution: Fixed
> Backport HBASE-25616 "Upgrade opentelemetry to 1.0.0" to branch-2
> -
>
> Key: HBASE-26135
> URL: https://issues.apache.org/jira/browse/HBASE-26135
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 12/17 commits of HBASE-22120, original commits
> [{{8399293}}|https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26136) Backport HBASE-25723 "Temporarily remove the trace support for RegionScanner.next" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26136.
--
Resolution: Fixed
commit pushed to
[https://github.com/apache/hbase/commit/817cfe1bf2c23de8cae178e0f96462ab5da33a59]
Thanks [~zhangduo] for reviewing.
> Backport HBASE-25723 "Temporarily remove the trace support for
> RegionScanner.next" to branch-2
> --
>
> Key: HBASE-26136
> URL: https://issues.apache.org/jira/browse/HBASE-26136
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 13/17 commits of HBASE-22120, original commit
> [{{7f90c22}}|https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26137) Backport HBASE-25732 "Change the command line argument for tracing after upgrading opentelemtry to 1.0.0" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26137.
--
Resolution: Fixed
commit pushed
https://github.com/apache/hbase/commit/d8f1ffa4d3731af654bf7ad94ee83dfa285fe168
> Backport HBASE-25732 "Change the command line argument for tracing after
> upgrading opentelemtry to 1.0.0" to branch-2
> -
>
> Key: HBASE-26137
> URL: https://issues.apache.org/jira/browse/HBASE-26137
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 14/17 commits of HBASE-22120, original commit
> [{{8df9beb}}|https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26138) Backport HBASE-25733 "Upgrade opentelemetry to 1.0.1" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26138.
--
Resolution: Fixed
> Backport HBASE-25733 "Upgrade opentelemetry to 1.0.1" to branch-2
> -
>
> Key: HBASE-26138
> URL: https://issues.apache.org/jira/browse/HBASE-26138
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 15/17 commits of HBASE-22120, original commit
> [{{b714889}}|https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26139) Backport HBASE-23762 "Add documentation on how to enable and view tracing with OpenTelemetry" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26139.
--
Hadoop Flags: Reviewed
Resolution: Fixed
[https://github.com/apache/hbase/commit/c96746736fa868d5a8859b501b6ab11e97f1a674]
pushed
> Backport HBASE-23762 "Add documentation on how to enable and view tracing
> with OpenTelemetry" to branch-2
> -
>
> Key: HBASE-26139
> URL: https://issues.apache.org/jira/browse/HBASE-26139
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 16/17 commits of HBASE-22120, original commit
> [{{be4503d}}|https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26226) Make RpcConnection with transient/non-Singleton SaslClientAuthenticationProviders with repsecting the passing configuration object
Tak-Lon (Stephen) Wu created HBASE-26226:
Summary: Make RpcConnection with transient/non-Singleton
SaslClientAuthenticationProviders with repsecting the passing configuration
object
Key: HBASE-26226
URL: https://issues.apache.org/jira/browse/HBASE-26226
Project: HBase
Issue Type: Improvement
Components: Client, rpc
Reporter: Tak-Lon (Stephen) Wu
When a RPC connection was created from any table connection (e.g. via
{{ConnectionFactory.createConnection(conf)}}), the
[SaslClientAuthenticationProviders|(https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/ipc/RpcConnection.java#L99-L101]
that matches different authentication selector per-user/TOKEN as the
[singleton
object|https://github.com/apache/hbase/blob/735bcf85e9b59a71981babe2f5da51978d61d8d3/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/SaslClientAuthenticationProviders.java#L72-L83]
and reuses for any followup connection creation.
If user want to create another connection with a different configuration key
that the authentication selector may use, e.g. [if one want to switch between
secure and non-secure
connection|https://github.com/apache/hbase/blob/master/hbase-client/src/main/java/org/apache/hadoop/hbase/security/provider/BuiltInProviderSelector.java#L109-L111],
it cannot be done because of the cached singleton instance of
AuthenticationProviders.
the goal of this JIRA is to allow the RpcConnection and AuthenticationProviders
align with whatever passing configuration object
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26140) Backport HBASE-25778 "The tracinig implementation for AsyncConnectionImpl.getHbck is incorrect" to branch-2
[
https://issues.apache.org/jira/browse/HBASE-26140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26140.
--
Hadoop Flags: Reviewed
Resolution: Fixed
https://github.com/apache/hbase/commit/d80763268621590b0701ec3fa2518fab6834d1b0
pushed
> Backport HBASE-25778 "The tracinig implementation for
> AsyncConnectionImpl.getHbck is incorrect" to branch-2
> ---
>
> Key: HBASE-26140
> URL: https://issues.apache.org/jira/browse/HBASE-26140
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> 17/17 commits of HBASE-22120, original commit
> [{{f36e153}}|https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a]
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26168) Backport HBASE-25811 for fixing nightly tests with error of `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
[
https://issues.apache.org/jira/browse/HBASE-26168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26168.
--
Hadoop Flags: Reviewed
Resolution: Fixed
> Backport HBASE-25811 for fixing nightly tests with error of
> `NoClassDefFoundError: io/opentelemetry/api/GlobalOpenTelemetry`
>
>
> Key: HBASE-26168
> URL: https://issues.apache.org/jira/browse/HBASE-26168
> Project: HBase
> Issue Type: Sub-task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> see nightly error
> [https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/313/artifact/output-integration/hadoop-2/mr-importtsv.err/*view*/]
> {code:java}
> Caused by: java.lang.NoClassDefFoundError:
> io/opentelemetry/api/GlobalOpenTelemetry
> at org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:33)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callMethod(AbstractRpcClient.java:398)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:328)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient.access$200(AbstractRpcClient.java:92)
> at
> org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:589)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$BlockingStub.get(ClientProtos.java:45761)
> at
> org.apache.hadoop.hbase.client.ClientServiceCallable.doGet(ClientServiceCallable.java:50)
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:377)
> at org.apache.hadoop.hbase.client.HTable$1.rpcCall(HTable.java:372)
> at
> org.apache.hadoop.hbase.client.RegionServerCallable.call(RegionServerCallable.java:127)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:108)
> ... 32 more
> Caused by: java.lang.ClassNotFoundException:
> io.opentelemetry.api.GlobalOpenTelemetry
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:418)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:355)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:351)
> ... 43 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26241) Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin
Tak-Lon (Stephen) Wu created HBASE-26241: Summary: Showcase/document a end-to-end tracing integreated with an external sink e.g. Jaeger or zipkin Key: HBASE-26241 URL: https://issues.apache.org/jira/browse/HBASE-26241 Project: HBase Issue Type: Task Affects Versions: 3.0.0-alpha-1, 2.5.0 Reporter: Tak-Lon (Stephen) Wu This task/issue was being discussed [in the vote of HBASE-25853|https://lists.apache.org/thread.html/rb36f0b6b179ee5f42cff7bb60de2797f69c450752c993f0b1e0f6118%40%3Cdev.hbase.apache.org%3E] bq. I've not seen anyone report their success at integrating these changes into a telemetry system and showing end-to-end tracing of a service call. As I recall from the last 2.5 discuss thread, we wanted this to be the banner feature for the minor release. Since we don't have this end-to-end confirmation, do we have someone who's volunteered to demonstrate that final proof-of-integration in a timely manner? Perhaps that someone is keen to be release manager for 2.5. What a `service call` mean ? (pending discussion on above email exchange or on this JIRA) e.g. for RPCclient/AbstractRpcClient, it trace at the AbstractRpcClient#callMethod level with a span of `RpcClient.callMethod` . or whenever a connection is closed, it will emit `AsyncConnection.close` -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (HBASE-25853) Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
[
https://issues.apache.org/jira/browse/HBASE-25853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-25853.
--
Resolution: Fixed
https://github.com/apache/hbase/pull/3637 merged into branch-2
> Backport HBASE-22120 (Replace HTrace with OpenTelemetry) to branch-2
>
>
> Key: HBASE-25853
> URL: https://issues.apache.org/jira/browse/HBASE-25853
> Project: HBase
> Issue Type: Task
> Components: tracing
>Affects Versions: 2.5.0
>Reporter: Duo Zhang
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> list of commits from upstream/master need to be backported
> {code}
> HBASE-25373
> https://github.com/apache/hbase/commit/302d9ea8b888762a5a20a5ba5c2be7bc239afaef
>
> HBASE-25401
> https://github.com/apache/hbase/commit/242028671535dfed67ec13d9ed0d6067f3ccfd04
>
> HBASE-25424
> https://github.com/apache/hbase/commit/57960fa8fa7228d65b1a4adc8e9b5b1a8158824d
>
> HBASE-23898
> https://github.com/apache/hbase/commit/805b2ae2ad0f6325515d46043ff01e4e2c7a9f59
>
> HBASE-25454
> https://github.com/apache/hbase/commit/dcb78bd4bda4a4ae13d863df8aec266031e5bc93
> HBASE-25481
> https://github.com/apache/hbase/commit/ae2c62ffaad5ba4c976b0a79c10a365edf2844fd
> HBASE-25455
> https://github.com/apache/hbase/commit/03e12bfa4ad62ecc6eee6a2c68d431bea2d5c473
> HBASE-25484
> https://github.com/apache/hbase/commit/2be2c63f0d3917a243b74af9754cbfc805b858d1
> HBASE-25535
> https://github.com/apache/hbase/commit/bb8c4967f8ce2c89ebaf1ddc5d8a1bf55f1e20d3
> HBASE-25591
> https://github.com/apache/hbase/commit/f6ff519dd0c7fe0e3ae3c175eefee27a26a065a4
> HBASE-25617
> https://github.com/apache/hbase/commit/8d68f8cd1c8613be1b499eaa99f46806b2743294
> HBASE-25616
> https://github.com/apache/hbase/commit/8399293e21127df3ffdcb757242e4cb5964c7e99
> HBASE-25723
> https://github.com/apache/hbase/commit/7f90c2201f6a17d2e2d031505c35ae7c2b1ed7ea
> HBASE-25732
> https://github.com/apache/hbase/commit/8df9bebdd367d52a32b08c18a7cf4f9c2d712071
> HBASE-25733
> https://github.com/apache/hbase/commit/b71488998970a3353086a34736ed1edab527f673
> HBASE-23762
> https://github.com/apache/hbase/commit/be4503d9f82f044fbfce21c8a42d0b1684607238
> HBASE-25778
> https://github.com/apache/hbase/commit/f36e1539648bbaee84c626fd54d1605baebf3c5a
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26273) TableSnapshotInputFormat/TableSnapshotInputFormatImpl should use ReadType.STREAM for scanning HFiles
Tak-Lon (Stephen) Wu created HBASE-26273:
Summary: TableSnapshotInputFormat/TableSnapshotInputFormatImpl
should use ReadType.STREAM for scanning HFiles
Key: HBASE-26273
URL: https://issues.apache.org/jira/browse/HBASE-26273
Project: HBase
Issue Type: Improvement
Components: mapreduce
Affects Versions: 2.4.6, 3.0.0-alpha-1
Reporter: Tak-Lon (Stephen) Wu
After the change in HBASE-17917 that use PREAD ({{ReadType.DEFAULT}}) for all
user scan, the behavior of TableSnapshotInputFormat changed from STREAM to
PREAD.
TableSnapshotInputFormat is supposed to be use with a YARN/MR or other batch
engine that should read the entire HFile in the container/executor, with
default always to PREAD, the number of connection to HDFS surges and has an
side-effect on the overall performance.
The goal of this change is to make any downstream using
TableSnapshotInputFormat with STREAM scan.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26274) Create an option to reintroduce BlockCache to mapreduce job
Tak-Lon (Stephen) Wu created HBASE-26274:
Summary: Create an option to reintroduce BlockCache to mapreduce
job
Key: HBASE-26274
URL: https://issues.apache.org/jira/browse/HBASE-26274
Project: HBase
Issue Type: Bug
Components: BlockCache, HFile, mapreduce
Affects Versions: 2.4.6, 3.0.0-alpha-1
Reporter: Tak-Lon (Stephen) Wu
In HBASE-21498 (see [this
commit|https://github.com/apache/hbase/commit/27a0f205c52f83fe7500ee2ffc6cf6582f565a63#diff-8a3e39e6df1afe47811fc17702da598fe0d80496d66e579bea4bd224c6d8da03R218],
it change the behavior that only region server can initialize on-heap
BlockCache/LruBlockCache, this should be the right change for HMaster.
Other downstream dependency that uses getScanner from a file-based region and
read HStore/HFile lost the BlockCache for caching INDEX/LEAF_INDEX (at least
still a problem with HBase-2.4) after this change (it worked before) and caused
performance impact with 2x slower.
One way to bring back the performance is to allow non-RS and non-HMaster can
use a compact version of blockcache with smaller memory and less hbase internal
configuration.
Or if we can find a way to cache or skip reading the same {{LEAF_INDEX}} when
scanning the DATA block with HFile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26274) Create an option to reintroduce BlockCache to mapreduce job
[
https://issues.apache.org/jira/browse/HBASE-26274?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26274.
--
Resolution: Fixed
> Create an option to reintroduce BlockCache to mapreduce job
> ---
>
> Key: HBASE-26274
> URL: https://issues.apache.org/jira/browse/HBASE-26274
> Project: HBase
> Issue Type: Bug
> Components: BlockCache, HFile, mapreduce
>Affects Versions: 3.0.0-alpha-1, 2.4.6
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0, 2.3.7, 2.4.7, 3.0.0-alpha-1
>
>
> In HBASE-21498 (see [this
> commit|https://github.com/apache/hbase/commit/27a0f205c52f83fe7500ee2ffc6cf6582f565a63#diff-8a3e39e6df1afe47811fc17702da598fe0d80496d66e579bea4bd224c6d8da03R218],
> it change the behavior that only region server can initialize on-heap
> BlockCache/LruBlockCache, this should be the right change for HMaster.
> Other downstream dependency that uses getScanner from a file-based region and
> read HStore/HFile lost the BlockCache for caching INDEX/LEAF_INDEX (at least
> still a problem with HBase-2.4) after this change (it worked before) and
> caused performance impact with 2x slower.
> One way to bring back the performance is to allow non-RS and non-HMaster can
> use a compact version of blockcache with smaller memory and less hbase
> internal configuration.
> Or if we can find a way to cache or skip reading the same {{LEAF_INDEX}} when
> scanning the DATA block with HFile.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26141) Add tracing support for HTable and sync connection on branch-2
[ https://issues.apache.org/jira/browse/HBASE-26141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26141. -- Hadoop Flags: Reviewed Resolution: Fixed > Add tracing support for HTable and sync connection on branch-2 > -- > > Key: HBASE-26141 > URL: https://issues.apache.org/jira/browse/HBASE-26141 > Project: HBase > Issue Type: New Feature > Components: tracing >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.5.0 > > > Split from HBASE-25853, the original change of HBASE-22120 does not include > tracing support for sync client, if we decide to support it in branch-2, we > will need to come up our own solution. > this change should cover mainly the classes of HTable, > ConnectionImplementation, and HRegionLocator > * RPC and IPC tracing supported by HBASE-26125 > * WAL tracing supported by HBASE-26131 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (HBASE-26299) Fix TestHTableTracing.testTableClose for nightly build of branch-2
Tak-Lon (Stephen) Wu created HBASE-26299:
Summary: Fix TestHTableTracing.testTableClose for nightly build of
branch-2
Key: HBASE-26299
URL: https://issues.apache.org/jira/browse/HBASE-26299
Project: HBase
Issue Type: Bug
Components: test, tracing
Affects Versions: 2.5.0
Reporter: Tak-Lon (Stephen) Wu
sometime isn't right with the last testTableClose when we close the table and
the connection, need to figure out why it's not working in the unit test.
{code}
[ERROR] org.apache.hadoop.hbase.client.TestHTableTracing.testTableClose Time
elapsed: 0.001 s <<< ERROR!
java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been
called. GlobalOpenTelemetry.set must be called only once before any calls to
GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use
OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation set
to cause of this exception.
at
io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:83)
at
io.opentelemetry.sdk.testing.junit4.OpenTelemetryRule.before(OpenTelemetryRule.java:95)
at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
at
org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
at
org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
at
org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
at
org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.Throwable
at
io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:91)
at
io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:61)
at
io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:110)
at
org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:71)
at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:95)
at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:78)
at
org.apache.hadoop.hbase.trace.TraceUtil.lambda$trace$1(TraceUtil.java:176)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:180)
at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:176)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.close(ConnectionImplementation.java:2110)
at
org.apache.hadoop.hbase.client.ConnectionImplementation.finalize(ConnectionImplementation.java:2149)
at java.lang.System$2.invokeFinalize(System.java:1273)
at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
{code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-26299) Fix TestHTableTracing.testTableClose for nightly build of branch-2
[
https://issues.apache.org/jira/browse/HBASE-26299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26299.
--
Fix Version/s: 2.5.0
Hadoop Flags: Reviewed
Resolution: Fixed
> Fix TestHTableTracing.testTableClose for nightly build of branch-2
> --
>
> Key: HBASE-26299
> URL: https://issues.apache.org/jira/browse/HBASE-26299
> Project: HBase
> Issue Type: Bug
> Components: test, tracing
>Affects Versions: 2.5.0
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> after merging HBASE-26141, sometime isn't right with the last testTableClose
> when we close the table and the connection, need to figure out why it's not
> working in the unit test with the jdk8 and hadoop 3 profile
> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/351/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/
> https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/branch-2/352/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/
> {code}
> [ERROR] org.apache.hadoop.hbase.client.TestHTableTracing.testTableClose Time
> elapsed: 0.001 s <<< ERROR!
> java.lang.IllegalStateException: GlobalOpenTelemetry.set has already been
> called. GlobalOpenTelemetry.set must be called only once before any calls to
> GlobalOpenTelemetry.get. If you are using the OpenTelemetrySdk, use
> OpenTelemetrySdkBuilder.buildAndRegisterGlobal instead. Previous invocation
> set to cause of this exception.
> at
> io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:83)
> at
> io.opentelemetry.sdk.testing.junit4.OpenTelemetryRule.before(OpenTelemetryRule.java:95)
> at org.junit.rules.ExternalResource$1.evaluate(ExternalResource.java:50)
> at org.junit.runners.ParentRunner$3.evaluate(ParentRunner.java:306)
> at
> org.junit.runners.BlockJUnit4ClassRunner$1.evaluate(BlockJUnit4ClassRunner.java:100)
> at org.junit.runners.ParentRunner.runLeaf(ParentRunner.java:366)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:103)
> at
> org.junit.runners.BlockJUnit4ClassRunner.runChild(BlockJUnit4ClassRunner.java:63)
> at org.junit.runners.ParentRunner$4.run(ParentRunner.java:331)
> at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:79)
> at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:329)
> at org.junit.runners.ParentRunner.access$100(ParentRunner.java:66)
> at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:293)
> at
> org.apache.hadoop.hbase.SystemExitRule$1.evaluate(SystemExitRule.java:38)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:288)
> at
> org.junit.internal.runners.statements.FailOnTimeout$CallableStatement.call(FailOnTimeout.java:282)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.lang.Throwable
> at
> io.opentelemetry.api.GlobalOpenTelemetry.set(GlobalOpenTelemetry.java:91)
> at
> io.opentelemetry.api.GlobalOpenTelemetry.get(GlobalOpenTelemetry.java:61)
> at
> io.opentelemetry.api.GlobalOpenTelemetry.getTracer(GlobalOpenTelemetry.java:110)
> at
> org.apache.hadoop.hbase.trace.TraceUtil.getGlobalTracer(TraceUtil.java:71)
> at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:95)
> at org.apache.hadoop.hbase.trace.TraceUtil.createSpan(TraceUtil.java:78)
> at
> org.apache.hadoop.hbase.trace.TraceUtil.lambda$trace$1(TraceUtil.java:176)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:180)
> at org.apache.hadoop.hbase.trace.TraceUtil.trace(TraceUtil.java:176)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.close(ConnectionImplementation.java:2110)
> at
> org.apache.hadoop.hbase.client.ConnectionImplementation.finalize(ConnectionImplementation.java:2149)
> at java.lang.System$2.invokeFinalize(System.java:1273)
> at java.lang.ref.Finalizer.runFinalizer(Finalizer.java:102)
> at java.lang.ref.Finalizer.access$100(Finalizer.java:34)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:217)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (HBASE-24833) Bootstrap should not delete the META table directory if it's not partial
[
https://issues.apache.org/jira/browse/HBASE-24833?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-24833.
--
Resolution: Fixed
> Bootstrap should not delete the META table directory if it's not partial
>
>
> Key: HBASE-24833
> URL: https://issues.apache.org/jira/browse/HBASE-24833
> Project: HBase
> Issue Type: Sub-task
>Affects Versions: 3.0.0-alpha-1, 2.3.0, 2.3.1, 2.3.3
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 2.5.0
>
>
> this issues were discussed in
> [PR#2113|https://github.com/apache/hbase/pull/2113] as part of HBASE-24286,
> and it is a dependencies before we solve HBASE-24286.
> The changes were introduced in [HBASE-24471
> |https://github.com/apache/hbase/commit/4d5efec76718032a1e55024fd5133409e4be3cb8#diff-21659161b1393e6632730dcbea205fd8R70-R89]
> that partial meta was introduced and `partial` was defined as
> InitMetaProcedure did not succeed and INIT_META_ASSIGN_META was not completed.
> {code:java}
> private static void writeFsLayout(Path rootDir, Configuration conf) throws
> IOException {
>LOG.info("BOOTSTRAP: creating hbase:meta region");
>FileSystem fs = rootDir.getFileSystem(conf);
>Path tableDir = CommonFSUtils.getTableDir(rootDir,
> TableName.META_TABLE_NAME);
>if (fs.exists(tableDir) && !fs.delete(tableDir, true)) {
> LOG.warn("Can not delete partial created meta table, continue...");
>}
> {code}
> however, in the cloud use case where HFiles store on S3, WALs store on HDFS,
> ZK data are stored within the cluster, this partial meta becomes a block when
> cluster recreate on existing HFiles; Here, Zk data and WALs cannot be
> retained (HDFS was associated with cloud instance and was terminated
> together) when cluster recreates on the flushed HFiles, and existing meta are
> always considered as partial and deleted in `INIT_META_WRITE_FS_LAYOUT`
> during bootstrap. As a result, the recreate cluster starts with a empty meta
> table, either the cluster hangs during the master initialization (branch-2)
> because table states of namespace table cannot be assigned, or starts as a
> fresh cluster without any region assigned and table opens (may need HBCK to
> rebuild the meta).
> Potential solution suggested by Anoop
> {quote}In case of HM start and the bootstrap we create the ClusterID and
> write to FS and then to zk and then create the META table FS layout. So in a
> cluster recreate, we will see clusterID is there in FS and also the META FS
> layout but no clusterID in zk. Ya seems we can use this as indication for
> cluster recreate over existing data. In HM start, this is some thing we need
> to check at 1st itself and track. If this mode is true, later when (if) we do
> INIT_META_WRITE_FS_LAYOUT , we should not delete the META dir. As part of the
> Bootstrap when we write that proc to MasterProcWal, we can include this mode
> (boolean) info also. This is a protobuf message anyways. So even if this HM
> got killed and restarted (at a point where the clusterId was written to zk
> but the Meta FS layout part was not reached) we can use the info added as
> part of the bootstrap wal entry and make sure NOT to delete the meta dir.
> {quote}
> In this JIRA, we're going to fix the `partial` definition when we found
> cluster ID was stored in HFiles but ZK were deleted or fresh on cluster
> creates.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (HBASE-26524) Support remove coprocessor by class name via alter table command
Tak-Lon (Stephen) Wu created HBASE-26524:
Summary: Support remove coprocessor by class name via alter table
command
Key: HBASE-26524
URL: https://issues.apache.org/jira/browse/HBASE-26524
Project: HBase
Issue Type: Improvement
Components: Coprocessors, shell
Affects Versions: 2.5.0, 3.0.0-alpha-2
Reporter: Tak-Lon (Stephen) Wu
With the shell, when operator wants to remove a table coprocessor, the flow is
to
1. first use {{decs}} find the the mapping of coprocessor$# e.g. coprocessor$1,
where # is the ordered number when the coprocessor was internally added to the
table attribute
2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
value that include the unique class name.
This task is to simplify the flow if the operator know exactly the class name
of the added coprocessor, and create a new sub-method to {{alter}}, such that
operator can do it only with the class name.
NOTE that this logic has been added behind the scenes at
[TableDescriptorBuilder#removeCoprocessor](https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565)
for removing `ConstraintProcessor` , and we are just exposing this logic with
a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26529) Add document for HBASE-26524 to book.html
Tak-Lon (Stephen) Wu created HBASE-26529:
Summary: Add document for HBASE-26524 to book.html
Key: HBASE-26529
URL: https://issues.apache.org/jira/browse/HBASE-26529
Project: HBase
Issue Type: Task
Components: Coprocessors, shell
Reporter: Tak-Lon (Stephen) Wu
HBASE-26524 has merged, and we need to update the documentation such that
operator knows there is a method of {{table_remove_coprocessor}} for {{alter}}
command
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Created] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
Tak-Lon (Stephen) Wu created HBASE-26530: Summary: Backport HBASE-26524 Support remove coprocessor by class name via alter table command Key: HBASE-26530 URL: https://issues.apache.org/jira/browse/HBASE-26530 Project: HBase Issue Type: Task Affects Versions: 2.5.0 Reporter: Tak-Lon (Stephen) Wu porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Implemented
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu reopened HBASE-26524:
--
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26529) Document HBASE-26524 to section of Dynamic Unloading
[
https://issues.apache.org/jira/browse/HBASE-26529?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26529.
--
Fix Version/s: 3.0.0-alpha-2
Hadoop Flags: Reviewed
Release Note:
added the basic command usage of `table_remove_coprocessor` to the definition
guide at src/main/asciidoc/_chapters/cp.adoc.
hbase> alter 'users', METHOD => 'table_remove_coprocessor', CLASSNAME =>
'org.myname.hbase.Coprocessor.RegionObserverExample'
Resolution: Fixed
> Document HBASE-26524 to section of Dynamic Unloading
>
>
> Key: HBASE-26529
> URL: https://issues.apache.org/jira/browse/HBASE-26529
> Project: HBase
> Issue Type: Task
> Components: Coprocessors, shell
>Affects Versions: 2.5.0, 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Minor
> Fix For: 3.0.0-alpha-2
>
>
> HBASE-26524 has merged, and we need to update the documentation such that
> operator knows there is a method of {{table_remove_coprocessor}} for
> {{alter}} command
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu resolved HBASE-26524.
--
Hadoop Flags: Incompatible change,Reviewed
Resolution: Fixed
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: incompatible, incompatibleChange
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Reopened] (HBASE-26524) Support remove coprocessor by class name via alter table command
[
https://issues.apache.org/jira/browse/HBASE-26524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tak-Lon (Stephen) Wu reopened HBASE-26524:
--
> Support remove coprocessor by class name via alter table command
>
>
> Key: HBASE-26524
> URL: https://issues.apache.org/jira/browse/HBASE-26524
> Project: HBase
> Issue Type: Improvement
> Components: Coprocessors, shell
>Affects Versions: 3.0.0-alpha-2
>Reporter: Tak-Lon (Stephen) Wu
>Assignee: Tak-Lon (Stephen) Wu
>Priority: Major
> Labels: incompatible, incompatibleChange
> Fix For: 3.0.0-alpha-2
>
>
> With the shell, when operator wants to remove a table coprocessor, the flow
> is to
> 1. first use {{decs}} find the the mapping of coprocessor$# e.g.
> coprocessor$1, where # is the ordered number when the coprocessor was
> internally added to the table attribute
> 2. issue {{table_att_unset}} with the target `coprocessor$#` that maps to a
> value that include the unique class name.
> This task is to simplify the flow if the operator know exactly the class name
> of the added coprocessor, and create a new sub-method to {{alter}}, such that
> operator can do it only with the class name.
> NOTE that this logic has been added behind the scenes at
> [TableDescriptorBuilder#removeCoprocessor|https://github.com/apache/hbase/blob/358c4dc9022c507ee0159c1d4916aba41d42cde8/hbase-client/src/main/java/org/apache/hadoop/hbase/client/TableDescriptorBuilder.java#L1537-L1565]
> for removing `ConstraintProcessor` , and we are just exposing this logic
> with a new method to {{alter}} command.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Resolved] (HBASE-26530) Backport HBASE-26524 Support remove coprocessor by class name via alter table command
[ https://issues.apache.org/jira/browse/HBASE-26530?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tak-Lon (Stephen) Wu resolved HBASE-26530. -- Hadoop Flags: Reviewed Resolution: Fixed > Backport HBASE-26524 Support remove coprocessor by class name via alter table > command > -- > > Key: HBASE-26530 > URL: https://issues.apache.org/jira/browse/HBASE-26530 > Project: HBase > Issue Type: Task > Components: Coprocessors, shell >Affects Versions: 2.5.0 >Reporter: Tak-Lon (Stephen) Wu >Assignee: Tak-Lon (Stephen) Wu >Priority: Major > Fix For: 2.6.0 > > > porting HBASE-26524 to branch-2 -- This message was sent by Atlassian Jira (v8.20.1#820001)
