[jira] [Created] (HBASE-28684) Remove CellWrapper and use ExtendedCell internally in client side data structure
Duo Zhang created HBASE-28684: - Summary: Remove CellWrapper and use ExtendedCell internally in client side data structure Key: HBASE-28684 URL: https://issues.apache.org/jira/browse/HBASE-28684 Project: HBase Issue Type: Sub-task Components: API, Client Reporter: Duo Zhang In general, all Cells in HBase are ExtendedCells, we introduce Cell interface is only for preventing user to call some methods which damage the system. So I think we should have internal methods which can get ExtendedCell from the client side data structures so we do not need to cast everywhere. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28644) Use ExtendedCell instead of Cell in KeyValueScanner
[ https://issues.apache.org/jira/browse/HBASE-28644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28644. --- Fix Version/s: 2.7.0 3.0.0-beta-2 Hadoop Flags: Reviewed Resolution: Fixed Pushed to branch-2+. Thanks [~apurtell] for reviewing and [~stoty] for helping fixing Phoenix! > Use ExtendedCell instead of Cell in KeyValueScanner > --- > > Key: HBASE-28644 > URL: https://issues.apache.org/jira/browse/HBASE-28644 > Project: HBase > Issue Type: Sub-task > Components: API, regionserver, Scanners >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 2.7.0, 3.0.0-beta-2 > > > Need to do this before removing some deprecated methods in Cell, as we still > need to use these method at server side. -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Resolved] (HBASE-28617) Add trademark statement in footer on our website
[ https://issues.apache.org/jira/browse/HBASE-28617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28617. --- Resolution: Fixed Change category back to http://projects.apache.org/category/database > Add trademark statement in footer on our website > > > Key: HBASE-28617 > URL: https://issues.apache.org/jira/browse/HBASE-28617 > Project: HBase > Issue Type: Task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Reopened] (HBASE-28617) Add trademark statement in footer on our website
[ https://issues.apache.org/jira/browse/HBASE-28617?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang reopened HBASE-28617: --- For addressing doap issues. > Add trademark statement in footer on our website > > > Key: HBASE-28617 > URL: https://issues.apache.org/jira/browse/HBASE-28617 > Project: HBase > Issue Type: Task > Components: website >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28683) Only allow one TableProcedureInterface for a single table to run at the same time for some special procedure types
Duo Zhang created HBASE-28683: - Summary: Only allow one TableProcedureInterface for a single table to run at the same time for some special procedure types Key: HBASE-28683 URL: https://issues.apache.org/jira/browse/HBASE-28683 Project: HBase Issue Type: Improvement Components: master, proc-v2 Reporter: Duo Zhang We have a table lock in the MasterProcedureScheduler, which is designed to only allow one procedure to run at the same time when they require exclusive lock. But there is a problem that for availability, usually we can not always hold the exclusive lock through the whole procedure life time, as if so, we can not execute region assignment for this table too. The solution is to set holdLock to false, which means we will release the table lock after one execution cycle. In this way, it is possible that different table procedures may execute at the same time, which could mess things up. Especially that, in HBASE-28522, we find out that it is even impossible for DisableTableProcedure to hold the exclusive lock all the time. If the steps for DisableTableProcedure can be overlapped with other procedures like ModifyTableProcedure or even EnableTableProcedure, things will be definationly messed up... So we need to find another way to ensure that for a single table, only one of these procedures can be executed at the same time. -- This message was sent by Atlassian Jira (v8.20.10#820010)
Re: [DISCUSS] About the cleanup of our Cell related methods
Since PHOENIX-7331 has been resolved, I will also merge the PR for HBASE-28644, Thanks all for helping moving forward here! Istvan Toth 于2024年6月17日周一 13:37写道： > > Hi Duo, > > Yes, it does: > > [ERROR] Failed to execute goal > org.apache.maven.plugins:maven-compiler-plugin:3.11.0:compile > (default-compile) on project phoenix-core-client: Compilation failure: > Compilation failure: > [ERROR] > /home/stoty/workspaces/apache-phoenix/phoenix/phoenix-core-client/src/main/java/org/apache/phoenix/util/PhoenixKeyValueUtil.java:[262,93] > incompatible types: java.util.List cannot be > converted to java.util.List > [ERROR] > /home/stoty/workspaces/apache-phoenix/phoenix/phoenix-core-client/src/main/java/org/apache/phoenix/hbase/index/util/IndexManagementUtil.java:[248,69] > incompatible types: java.util.List cannot be > converted to java.util.List > > Now, we haven't actually released (TBH we haven't even committed it) Hbase > 2.6 support yet, so technically this doesn't break any released Phoenix > code. > > I played around with it a bit, and it looks like those two errors can be > fixed in a backwards-compatible manner with some work. > > Opened https://issues.apache.org/jira/browse/PHOENIX-7331 to track that > work. > > > On Mon, Jun 17, 2024 at 3:12 AM 张铎(Duo Zhang) wrote: > > > Hi, Istvan, do you have any findings? Does this break Phoenix on > > branch-2.6? > > > > Thanks. > > > > 张铎(Duo Zhang) 于2024年6月14日周五 09:32写道： > > > > > > The PR for branch-2 > > > > > > https://github.com/apache/hbase/pull/5985 > > > > > > The PR for branch-2.6 > > > > > > https://github.com/apache/hbase/pull/5990 > > > > > > At least the UTs are all fine. > > > > > > The current PR does not do much incompatible change, the only extra > > > check is we require filters to return ExtendedCell instead of Cell. > > > > > > Thanks. > > > > > > 张铎(Duo Zhang) 于2024年6月13日周四 14:33写道： > > > > > > > > OK, let me open PRs for branch-2 and branch-2.6 too. > > > > > > > > Istvan Toth 于2024年6月13日周四 14:02写道： > > > > > > > > > > I am also concerned about this change on branch-2. > > > > > Some of our code does implement getSequenceId() , so doing this on > > 2.5 and > > > > > 2.6 would definitely break the existing releases, and require some > > changes > > > > > and new compatibility modules on the Phoenix side. > > > > > > > > > > Phoenix usually uses the KeyValue type internally, which already > > extends > > > > > ExtendedCell, so it's probably not a huge problem, but we'd like to > > check > > > > > before this is merged to branch-2. > > > > > I'd like to ask for a heads-up when we have a PR for branch-2 so > > that we > > > > > can check what breaks. > > > > > (We currently only support branch-2.6, so ideally we'd have a > > branch-2.6 PR > > > > > to check, but we can probably get branch-2 to work to check the > > change > > > > > without a lot of work) > > > > > > > > > > We also have a POC branch for HBase 3, so we can also check Phoenix > > with a > > > > > branch-3 version of this change. > > > > > > > > > > The worst case scenario for Phoenix would be having to open a > > different > > > > > branch for hbase 3.x support, we really want to avoid that. > > > > > As long as the change can be papered over with a compatibility > > module, > > > > > we're mostly fine. > > > > > > > > > > Istvan > > > > > > > > > > > > > > > > > > > > On Thu, Jun 13, 2024 at 5:45 AM 张铎(Duo Zhang) > > wrote: > > > > > > > > > > > In fact, I do not think CPs can really return their own Cell > > > > > > implementation which is not an ExtendedCell, we will call > > > > > > PrivateCellUtil.setSequenceId and setTimestamp at server side, if > > the > > > > > > Cell is not an ExtendedCell we will throw IOException... > > > > > > > > > > > > I guess the only place where it is safe to return a customized > > Cell is > > > > > > in filter implementation, in getNextCellHint, where we will only > > use > > > > > > it to seek the scanners without storing it anywhere... > > > > > > > > > > > > Reid Chan 于2024年6月13日周四 10:56写道： > > > > > > > > > > > > > > It may affect Phoenix, as it is CPs related, at least for > > branch-2. > > > > > > > > > > > > > > Let's wait and see if there are any comments from them > > > > > > > > > > > > > > > > > > > > > --- > > > > > > > > > > > > > > Best regards, > > > > > > > R.C > > > > > > > > > > > > > > > > > > > > > > > > > > > > On Tue, Jun 11, 2024 at 4:19 PM Duo Zhang > > wrote: > > > > > > > > > > > > > > > We have several deprecated methods in Cell interface, like > > > > > > > > getSequenceId and tag related ones, which are marked as > > deprecated > > > > > > > > since 2.0.0 and should be removed in 3.0.0. Most of them are > > marked as > > > > > > > > deprecated in HBASE-19112. > > > > > > > > > > > > > > > > After investigating, I found that it is not an easy work... > > > > > > > > > > > > > > > > We have 3 levels for the Cell interface > > > > > > > > > > > > > > > > Cell -> RawCell -> ExtendedCell > > > > > > > > > > > > >
[jira] [Resolved] (HBASE-28548) Add documentation about the URI based connection registry
[ https://issues.apache.org/jira/browse/HBASE-28548?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Duo Zhang resolved HBASE-28548. --- Fix Version/s: 4.0.0-alpha-1 Hadoop Flags: Reviewed Resolution: Fixed Merged to master. Thanks [~ndimiduk] for reviewing! > Add documentation about the URI based connection registry > - > > Key: HBASE-28548 > URL: https://issues.apache.org/jira/browse/HBASE-28548 > Project: HBase > Issue Type: Sub-task > Components: Client, documentation >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Labels: pull-request-available > Fix For: 4.0.0-alpha-1 > > -- This message was sent by Atlassian Jira (v8.20.10#820010)
[jira] [Created] (HBASE-28682) ITBLL and other MR-based integration tests should heartbeat often
Nick Dimiduk created HBASE-28682:
Summary: ITBLL and other MR-based integration tests should
heartbeat often
Key: HBASE-28682
URL: https://issues.apache.org/jira/browse/HBASE-28682
Project: HBase
Issue Type: Test
Components: integration tests, mapreduce
Reporter: Nick Dimiduk
We have this little note in our ITBLL harness,
{noformat}
// If we cause enough chaos, RPC requests might get into long backoffs.
During this
// time, it won't send keep alives to the map/reduce context. So increase
the timeout
// a bunch
{noformat}
Investigating, the ITBLL Generator's persist method updates the MR context
progress only every 100 puts. You'd think that would be enough, but given
chaos, it really isn't. What if we update progress with every put? Digging
through MR source code, it seems that calling the context.progress() method
only sets an AtomicBoolean that a progress update needs sent, actual sending of
progress reports is gated by {{mapreduce.task.progress-report.interval}}, or 1%
of {{mapreduce.task.timeout}}, which defaults to 1% of 300_000ms, or 3 seconds.
So yeah, we should probably update this AtomicBool much more often in chaotic
jobs, as doing so is effectively free and will improve reliability.
But still, every put is perhaps excessive. What if we add a pre-flush hook to
(Async)BufferedMutator so that a MR job can set this progress flag right before
the client disappears down into a retry loop? I bet other applications would
find such a hook useful as well.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
