[jira] [Resolved] (HBASE-28464) Make replication ZKWatcher config customizable in extensions

[Szabolcs Bukros (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-28464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Bukros resolved HBASE-28464.
-
Resolution: Implemented

Implemented by HBASE-28529

> Make replication ZKWatcher config customizable in extensions
> 
>
> Key: HBASE-28464
> URL: https://issues.apache.org/jira/browse/HBASE-28464
> Project: HBase
>  Issue Type: Improvement
>  Components: Replication
>        Reporter: Szabolcs Bukros
>    Assignee: Szabolcs Bukros
>Priority: Major
>  Labels: pull-request-available
>
> The ZKWatcher in HBaseReplicationEndpoint always uses the source cluster's 
> ZooKeeper client config when connecting to the target cluster's zk. Those 
> might not match. I would like to make the used ZKClientConfig customizable 
> for replication extensions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-28464) Make replication ZKWatcher config customizable in extensions

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-28464:
---

 Summary: Make replication ZKWatcher config customizable in 
extensions
 Key: HBASE-28464
 URL: https://issues.apache.org/jira/browse/HBASE-28464
 Project: HBase
  Issue Type: Improvement
  Components: Replication
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


The ZKWatcher in HBaseReplicationEndpoint always uses the source cluster's 
ZooKeeper client config when connecting to the target cluster's zk. Those might 
not match. I would like to make the used ZKClientConfig customizable for 
replication extensions.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

[Szabolcs Bukros]

I would be fine with adding this only to 2.6 and 3.0. I only proposed the
other branches because of the perceived low impact of the change.
This change is not important enough to make a sooner release necessary.

If the change is accepted could I please get a review for the PR?
https://github.com/apache/hbase/pull/4885

On Wed, Jan 4, 2023 at 3:17 AM 张铎(Duo Zhang)  wrote:

> +1 on releasing 2.6.0 sooner.
>
> And I think it is time to EOL 2.4.x after we release 2.6.0?
>
> Bryan Beaudreault  于2023年1月3日周二 21:02写道：
> >
> > I think development is done on TLS. We are just waiting on requested
> > testing. Andor was working on that, but I believe he had some stuff come
> up
> > at his work.
> >
> > I also want to get backups in place, but there is 1 backwards
> compatibility
> > issue to work through. Hoping to have that squared away soon.
> >
> > On Sat, Dec 31, 2022 at 9:32 PM Andrew Purtell  >
> > wrote:
> >
> > > +1
> > >
> > > If this is needed soon in a release we could start on 2.6.0?
> > >
> > > (How is TLS RPC coming along? - that would be the big ticket item.)
> > >
> > > > On Dec 23, 2022, at 7:06 AM, 张铎  wrote:
> > > >
> > > > This is a behavior change, it makes non admin users can clone
> snapshot.
> > > >
> > > > For me I do not think we should include changes like this in a patch
> > > > release, unless it is considered as a critical bug which must be
> > > > fixed.
> > > >
> > > > Thanks.
> > > >
> > > > Szabolcs Bukros  于2022年11月30日周三
> 00:06写道：
> > > >>
> > > >> This should not break any existing use case so I see no reason to
> not
> > > add
> > > >> this to branch-2.5 and
> > > >> branch-2.4.
> > > >>
> > > >>> On Thu, Nov 24, 2022 at 3:03 AM 张铎(Duo Zhang) <
> palomino...@gmail.com>
> > > wrote:
> > > >>>
> > > >>> I'm OK with this change.
> > > >>>
> > > >>> But maybe we still need to determine which branches we can apply
> this
> > > >>> change to? Is it OK to include this change for branch-2.5 and
> > > >>> branch-2.4?
> > > >>>
> > > >>> Tak Lon (Stephen) Wu  于2022年11月22日周二 06:31写道：
> > > >>>>
> > > >>>> FYI the PR is https://github.com/apache/hbase/pull/4885
> > > <https://github.com/apache/hbase/pull/4885>
> > > and
> > > >>>> https://issues.apache.org/jira/browse/HBASE-27493
> > > <https://issues.apache.org/jira/browse/HBASE-27493>
> > > .
> > > >>>>
> > > >>>> the proposal seems to be, should we allow cloning snapshot to any
> > > >>>> namespace if they're not the global admin.
> > > >>>>
> > > >>>> logically, it should be fine because they're the admin for the
> > > >>>> namespace, and should be able to do whatever within that
> namespace.
> > > >>>>
> > > >>>> Thanks,
> > > >>>> Stephen
> > > >>>>
> > > >>>>
> > > >>>> On Mon, Nov 21, 2022 at 11:38 AM Szabolcs Bukros
> > > >>>>  wrote:
> > > >>>>>
> > > >>>>> Hi Everyone,
> > > >>>>>
> > > >>>>> Creating a snapshot requires table admin permissions. But
> cloning it
> > > >>>>> requires global admin permissions unless the user owns the
> snapshot
> > > and
> > > >>>>> wants to recreate the original table the snapshot was based on
> using
> > > >>> the
> > > >>>>> same table name. This puts unnecessary load on the few users
> having
> > > >>> global
> > > >>>>> admin permissions on the cluster. I would like to relax this
> rule a
> > > >>> bit and
> > > >>>>> allow the owner of the snapshot to clone it into any namespace
> where
> > > >>> they
> > > >>>>> have admin permissions regardless of the table name used.
> > > >>>>>
> > > >>>>> Please let me know what you think about this proposal. And if you
> > > find
> > > >>> it
> > > >>>>> acceptable which branch do you think this could land on.
> > > >>>>>
> > > >>>>> Thanks,
> > > >>>>> Szabolcs Bukros
> > > >>>
> > >
>


-- 
*Szabolcs Bukros* | Staff Engineer
e. szabo...@cloudera.com
cloudera.com <https://www.cloudera.com>
[image: Cloudera] <https://www.cloudera.com/>
[image: Cloudera on Twitter] <https://twitter.com/cloudera> [image:
Cloudera on Facebook] <https://www.facebook.com/cloudera> [image: Cloudera
on LinkedIn] <https://www.linkedin.com/company/cloudera>
<https://www.cloudera.com/>
--

Re: [DISCUSS] Allow namespace admins to clone snapshots created by them

[Szabolcs Bukros]

This should not break any existing use case so I see no reason to not add
this to branch-2.5 and
branch-2.4.

On Thu, Nov 24, 2022 at 3:03 AM 张铎(Duo Zhang)  wrote:

> I'm OK with this change.
>
> But maybe we still need to determine which branches we can apply this
> change to? Is it OK to include this change for branch-2.5 and
> branch-2.4?
>
> Tak Lon (Stephen) Wu  于2022年11月22日周二 06:31写道：
> >
> > FYI the PR is https://github.com/apache/hbase/pull/4885 and
> > https://issues.apache.org/jira/browse/HBASE-27493.
> >
> > the proposal seems to be, should we allow cloning snapshot to any
> > namespace if they're not the global admin.
> >
> > logically, it should be fine because they're the admin for the
> > namespace, and should be able to do whatever within that namespace.
> >
> > Thanks,
> > Stephen
> >
> >
> > On Mon, Nov 21, 2022 at 11:38 AM Szabolcs Bukros
> >  wrote:
> > >
> > > Hi Everyone,
> > >
> > > Creating a snapshot requires table admin permissions. But cloning it
> > > requires global admin permissions unless the user owns the snapshot and
> > > wants to recreate the original table the snapshot was based on using
> the
> > > same table name. This puts unnecessary load on the few users having
> global
> > > admin permissions on the cluster. I would like to relax this rule a
> bit and
> > > allow the owner of the snapshot to clone it into any namespace where
> they
> > > have admin permissions regardless of the table name used.
> > >
> > > Please let me know what you think about this proposal. And if you find
> it
> > > acceptable which branch do you think this could land on.
> > >
> > > Thanks,
> > > Szabolcs Bukros
>

[DISCUSS] Allow namespace admins to clone snapshots created by them

[Szabolcs Bukros]

Hi Everyone,

Creating a snapshot requires table admin permissions. But cloning it
requires global admin permissions unless the user owns the snapshot and
wants to recreate the original table the snapshot was based on using the
same table name. This puts unnecessary load on the few users having global
admin permissions on the cluster. I would like to relax this rule a bit and
allow the owner of the snapshot to clone it into any namespace where they
have admin permissions regardless of the table name used.

Please let me know what you think about this proposal. And if you find it
acceptable which branch do you think this could land on.

Thanks,
Szabolcs Bukros

[jira] [Created] (HBASE-27493) Allow namespace admins to clone snapshots created by them

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-27493:
---

 Summary: Allow namespace admins to clone snapshots created by them
 Key: HBASE-27493
 URL: https://issues.apache.org/jira/browse/HBASE-27493
 Project: HBase
  Issue Type: Improvement
  Components: snapshots
Affects Versions: 2.5.1, 3.0.0-alpha-3
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


Creating a snapshot requires table admin permissions. But cloning it requires 
global admin permissions unless the user owns the snapshot and wants to 
recreate the original table the snapshot was based on using the same table 
name. This puts unnecessary load on the few people having global admin 
permissions. I would like to relax this rule a bit and allow the owner of the 
snapshot to clone it into any namespace where they have admin permissions 
regardless of the table name used.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

[jira] [Created] (HBASE-27154) Backport missing MOB related changes to branch-2

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-27154:
---

 Summary: Backport missing MOB related changes to branch-2
 Key: HBASE-27154
 URL: https://issues.apache.org/jira/browse/HBASE-27154
 Project: HBase
  Issue Type: Bug
  Components: mob
Affects Versions: 2.6.0
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


While trying to backport https://issues.apache.org/jira/browse/HBASE-26969 to 
branch-2 I have found that multiple major MOB related changes are missing. 

This change is required for FileBased SFT correctness so the changes it depends 
on should be backported first. Also any improvement to MOB stability is usually 
welcomed.

The missing changes I have found so far:
https://issues.apache.org/jira/browse/HBASE-22749
https://issues.apache.org/jira/browse/HBASE-23723
https://issues.apache.org/jira/browse/HBASE-24163

There is also a docs change describing the new MOB functionality. But 
considering that the book is always generated based on master I think it is 
safe to skip backporting it.
https://issues.apache.org/jira/browse/HBASE-23198

I'm planning to backport these changes one by one until we reach a state where 
HBASE-26969  can be backported too.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HBASE-27017) MOB snapshot is broken when FileBased SFT is used

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-27017:
---

 Summary: MOB snapshot is broken when FileBased SFT is used
 Key: HBASE-27017
 URL: https://issues.apache.org/jira/browse/HBASE-27017
 Project: HBase
  Issue Type: Bug
  Components: mob
Affects Versions: 3.0.0-alpha-2, 2.5.0
Reporter: Szabolcs Bukros


During snapshot MOB regions are treated like any other region. When a snapshot 
is taken and hfile references are collected a StoreFileTracker is created to 
get the current active hfile list. But the MOB region stores are not tracked so 
an empty list is returned, resulting in a broken snapshot. When this snapshot 
is cloned the resulting table will have no MOB files or references.

The problematic code can be found here:

[https://github.com/apache/hbase/blob/master/hbase-server/src/main/java/org/apache/hadoop/hbase/snapshot/SnapshotManifest.java#L313]



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HBASE-26969) Eliminate MOB renames when SFT is enabled

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-26969:
---

 Summary: Eliminate MOB renames when SFT is enabled
 Key: HBASE-26969
 URL: https://issues.apache.org/jira/browse/HBASE-26969
 Project: HBase
  Issue Type: Task
  Components: mob
Affects Versions: 2.5.0, 3.0.0-alpha-3
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


MOB file compaction and flush still relies on renames even when SFT is enabled.

My proposed changes are:
 * when requireWritingToTmpDirFirst is false during mob flush/compact instead 
of using the temp writer we should create a different writer using a 
{color:#00}StoreFileWriterCreationTracker that writes directly to the mob 
store folder{color}
 * {color:#00}these StoreFileWriterCreationTracker should be stored in the 
MobStore. This would requires us to extend MobStore with a createWriter and a 
finalizeWriter method to handle this{color}
 * {color:#00}refactor {color}MobFileCleanerChore to run on the RS instead 
on Master to allow access to the 
{color:#00}StoreFileWriterCreationTracker{color}s to make sure the 
currently written files are not cleaned up



--
This message was sent by Atlassian Jira
(v8.20.7#820007)

[jira] [Created] (HBASE-26791) Memstore flush fencing issue for SFT

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-26791:
---

 Summary: Memstore flush fencing issue for SFT
 Key: HBASE-26791
 URL: https://issues.apache.org/jira/browse/HBASE-26791
 Project: HBase
  Issue Type: Bug
Affects Versions: 2.6.0, 3.0.0-alpha-3
Reporter: Szabolcs Bukros


The scenarios is the following:
 # rs1 is flushing file to S3 for region1
 # rs1 loses ZK lock
 # region1 gets assigned to rs2
 # rs2 opens region1
 # rs1 completes flush and updates sft file for region1
 # rs2 has a different “version” of the sft file for region1

The flush should fail at the end, but the SFT file gets overwritten before 
that, resulting in potential data loss.

 

Potential solutions include:
 * Adding timestamp to the tracker file names. This and creating a new tracker 
file when an rs open the region would allow us to list available tracker files 
before an update and compare the found timestamps to the one stored in memory 
to verify the store still owns the latest tracker file
 * Using the existing timestamp in the tracker file content. This would also 
require us to create a new tracker file when a new rs opens the region, but 
instead of listing the available tracker files, we could try to load and 
de-serialize the last tracker file and compare the timestamp found in it to the 
one stored in memory.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[DISCUSS] operator tools, HBase 3 and StoreFileTracking

[Szabolcs Bukros]

Hi Folks!

While working on adding tools to handle potential FileBased
StoreFileTracker issues to HBCK2 (HBASE-26624
) I ran into multiple
problems I'm unsure how to solve.

First of all the tools would rely on files not yet available in any of the
released hbase artifacts. I tried to solve this without changing the hbase
dependency version to keep HBCK2 as hbase version independent as possible,
but none of the solutions I have found looked acceptable:
 - Pushing the logic to the hbase side (as far as I can tell) is not
feasible because it has to be able to repair meta which is easier when
hbase is down and the tool should be able to run without a working hbase.
 - The files tracking the store content are serialized proto objects so
while replicating those files in the operator tools is possible, it would
not be pretty.

Bumping operator tools to use hbase 2.6.0-SNAPSHOT (branch-2 has the SFT
changes) would mean that now we need that or a newer version to build the
project and a version check to avoid runtime problems with the new tools,
but otherwise this looks rather painless and backwards compatible. I know
operator tools tries to avoid having a hbase-specific release, but having
2.6 as a min version to build against might be acceptable.

While looking into this I also checked what needs to be done to make
operator tools work with hbase 3.0.0-alpha-3-SNAPSHOT. Most of the changes
are backwards compatible but not all of them and the ones that aren't would
make a big chunk of Fsck unusable with older hbases. For me that looks
acceptable since this is a major version change, but that would mean I can
not rely on a potential HBCK3 to fix SFT issues, I would also need a
solution for HBCK2.

I tried to look for plans/direction regarding the new 1.3 operator tools
but could not find any.

Do you think it would be possible to bump the hbase version it uses to
2.6.0-SNAPSHOT?
Do you think it would make sense to start working on a hbase3 compatible
branch or is it too early?

NOTE:
I'm aware hbase does not publish SNAPSHOT builds for years, but I do not
know how the internal build system works and if these artifacts would be
available for internal builds or not. I also do not know if necessary could
they be made available.

[jira] [Created] (HBASE-26707) Reduce number of renames during bulkload

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-26707:
---

 Summary: Reduce number of renames during bulkload
 Key: HBASE-26707
 URL: https://issues.apache.org/jira/browse/HBASE-26707
 Project: HBase
  Issue Type: Sub-task
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


Make sure we only do a single rename operation during bulkload when StoreEngine 
does not require the the use of tmp directories.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-26441) Add metrics for BrokenStoreFileCleaner

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-26441:
---

 Summary: Add metrics for BrokenStoreFileCleaner
 Key: HBASE-26441
 URL: https://issues.apache.org/jira/browse/HBASE-26441
 Project: HBase
  Issue Type: Sub-task
  Components: metrics
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


This is a followup for HBASE-26271.
Cleaner chores lacking visibility is returning issue so I would like to add 
metrics for BrokenStoreFileCleaner to have a better idea of the tasks it 
performs.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (HBASE-25964) [HBOSS] Introducing hbase metrics to Hboss

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-25964:
---

 Summary: [HBOSS] Introducing hbase metrics to Hboss
 Key: HBASE-25964
 URL: https://issues.apache.org/jira/browse/HBASE-25964
 Project: HBase
  Issue Type: Improvement
  Components: hboss
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros
 Fix For: hbase-filesystem-1.0.0-alpha2


I would like to introduce hbase metrics to Hboss to allow closer monitoring of 
rename performance.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24720) Meta replicas not cleaned when disabled

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-24720:
---

 Summary: Meta replicas not cleaned when disabled
 Key: HBASE-24720
 URL: https://issues.apache.org/jira/browse/HBASE-24720
 Project: HBase
  Issue Type: Bug
  Components: read replicas
Affects Versions: 2.2.5, 3.0.0-alpha-1, 2.3.0, 2.4.0
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


The assignMetaReplicas method works kinda like this:
{code:java}
void assignMetaReplicas(){
if (numReplicas <= 1) return;

//create if needed then assign meta replicas

unassignExcessMetaReplica(numReplicas);
}
{code}
Now this unassignExcessMetaReplica method is the one that gets rid of the 
replicas we no longer need. It closes them and deletes their zNode.  
Unfortunately this only happens if we decreased the replica number. If we 
disabled it, by setting the replica number to 1 assignMetaReplicas returns 
instantly without cleaning up the no longer needed replicas resulting in 
replicas lingering around.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24579) Failed SASL authentication does not result in an exception on client side

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-24579:
---

 Summary: Failed SASL authentication does not result in an 
exception on client side
 Key: HBASE-24579
 URL: https://issues.apache.org/jira/browse/HBASE-24579
 Project: HBase
  Issue Type: Bug
  Components: rpc
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


When HBaseSaslRpcClient.saslConnect tries to authenticate it only reads the 
input stream if the process is not complete yet. However if the authentication 
failed and the process is completed the exception sent back in the stream never 
gets read.

We should always try to read the input stream even if the process is complete 
to make sure it was empty.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24562) Stabilize master startup with meta replicas enabled

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-24562:
---

 Summary: Stabilize master startup with meta replicas enabled
 Key: HBASE-24562
 URL: https://issues.apache.org/jira/browse/HBASE-24562
 Project: HBase
  Issue Type: Improvement
  Components: meta, read replicas
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


This is related to HBASE-21624 . 

I created a separate ticket because in the original one a "complete solution 
for meta replicas" was requested and this is not one. I'm just trying to make 
master startup more stable by making assigning meta replicas asynchronous and 
preventing a potential assignment failure from crashing master.

The idea is that starting master with less or even no meta replicas assigned is 
preferable to not having a running master.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-24186) RegionMover ignores replicationId

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-24186:
---

 Summary: RegionMover ignores replicationId
 Key: HBASE-24186
 URL: https://issues.apache.org/jira/browse/HBASE-24186
 Project: HBase
  Issue Type: Bug
  Components: read replicas
Affects Versions: master
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


When RegionMover looks up which rs hosts a region, it does this based on 
startRowKey. When read replication is enabled this might not return the 
expected region's data and this can prevent the moving of these regions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23995) Snapshoting a splitting region results in corrupted snapshot

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23995:
---

 Summary: Snapshoting a splitting region results in corrupted 
snapshot
 Key: HBASE-23995
 URL: https://issues.apache.org/jira/browse/HBASE-23995
 Project: HBase
  Issue Type: Bug
  Components: snapshots
Affects Versions: 2.0.2
Reporter: Szabolcs Bukros


The problem seems to originate from the fact that while the region split itself 
runs in a lock, the compactions following it run in separate threads. 
Alternatively the use of space quota policies can prevent compaction after a 
split and leads to the same issue.

In both cases the resulting snapshot will keep the split status of the parent 
region, but do not keep the references to the daughter regions, because they 
(splitA, splitB qualifiers) are stored separately in the meta table and do not 
propagate with the snapshot.

This is important because the in the freshly cloned table CatalogJanitor will 
find the parent region, realizes it is in split state, but because it can not 
find the daughter region references (haven't propagated) assumes parent could 
be cleaned up and deletes it. The archived region used in the snaphost only has 
back reference to the now also archived parent region and if the snapshot is 
deleted they both gets cleaned up. Unfortunately the daughter regions only 
contains hfile links, so at this point the data is lost.

How to reproduce:
{code:java}
hbase shell <

Re: [DISCUSS] Stop flagging Read Replication from next release

[Szabolcs Bukros]

Hi Nick,

Thank you for your answer!

"I'm not quite comfortable with removing the warning yet (more testing to be
done), but I'm hoping to get to that point, at least for our production
workloads, if not for 2.3.0, within the early releases of 2.3.x."

What would make you comfortable?
Do you have specific criteria in mind? Are there test results you would
like to see? Use of different load, bigger cluster or longer runtime?

I could try to provide these if you would let me know what you are looking
for.

Thanks,
Szabolcs

On Fri, Feb 7, 2020 at 12:03 AM Nick Dimiduk  wrote:

> Hi Szabolcs,
>
> Looks like that note was dropped in via HBASE-20830, about 2 years back.
> From what we've seen on our clusters, kicking the tires with branch-2.2 and
> branch-2, that was indeed the case. Based on that recent experience and
> that of others', there's a number of fixes for Procedures and hbck2 that
> better handle read replica regions. Try this git incantation [0] and look
> for issues related to "unknown servers" and "RIT".
>
> I'm not quite comfortable with removing the warning yet (more testing to be
> done), but I'm hoping to get to that point, at least for our production
> workloads, if not for 2.3.0, within the early releases of 2.3.x.
>
> Maybe someone else who's using and abusing this feature on a branch-2
> release could add their experience?
>
> Thanks,
> Nick
>
> [0]: git log --oneline branch-2 --
> hbase-server/src/main/java/org/apache/hadoop/hbase/master/assignment/ |
> head -n20
>
> On Thu, Feb 6, 2020 at 1:26 AM Szabolcs Bukros
>  wrote:
>
> > Hi!
> >
> > In the current Reference Guide the section "75. Timeline-consistent High
> > Available Reads" is flagged as "maybe broken. Use it with caution". I'm
> not
> > familiar with the original reason it was flagged but I have spent a few
> > weeks working on this and after a few small fixes it looked stable
> enough.
> > I think we should remove this warning for new 2.2+ releases. Below are
> some
> > details about the fixes and the testing I did.
> >
> > Fixes:
> > - HBASE-23589 FlushDescriptor contains non-matching family/output
> > combinations <https://issues.apache.org/jira/browse/HBASE-23589>
> > - HBASE-23601 OutputSink.WriterThread exception gets stuck and repeated
> > indefinitely <https://issues.apache.org/jira/browse/HBASE-23601>
> >
> > Testing:
> > After the fixes I run IntegrationTestRegionReplicaReplication for testing
> > on a 4 machine cluster (3 RS, 30GB heap/RS). I used the default test
> > parameters, only increased read_delay_ms to 6. The longest
> > uninterrupted run I tried was 8 hours and I encountered no issues. Even
> > adding in the chaos monkeys (slowDeterministic) hasn't revealed any new
> > correctness issues with the feature.
> >
> > Next steps:
> > - Further testing. I realize IntegrationTestRegionReplicaReplication
> > provides a very uniform, unrealistic load, using different data could be
> > interesting. If someone would find the time to run a few tests or propose
> > some scenarios I would be grateful.
> > - I was thinking of providing a cleaner flush logic on replication side,
> > but my proposal might have too much overhead and the current logic while
> > having issues works after the previous fixes. The proposal can be found
> in
> > HBASE-23591, any feedback would be welcomed.
> >
> > Thoughts?
> >
>

--

[DISCUSS] Stop flagging Read Replication from next release

[Szabolcs Bukros]

Hi!

In the current Reference Guide the section "75. Timeline-consistent High
Available Reads" is flagged as "maybe broken. Use it with caution". I'm not
familiar with the original reason it was flagged but I have spent a few
weeks working on this and after a few small fixes it looked stable enough.
I think we should remove this warning for new 2.2+ releases. Below are some
details about the fixes and the testing I did.

Fixes:
- HBASE-23589 FlushDescriptor contains non-matching family/output
combinations 
- HBASE-23601 OutputSink.WriterThread exception gets stuck and repeated
indefinitely 

Testing:
After the fixes I run IntegrationTestRegionReplicaReplication for testing
on a 4 machine cluster (3 RS, 30GB heap/RS). I used the default test
parameters, only increased read_delay_ms to 6. The longest
uninterrupted run I tried was 8 hours and I encountered no issues. Even
adding in the chaos monkeys (slowDeterministic) hasn't revealed any new
correctness issues with the feature.

Next steps:
- Further testing. I realize IntegrationTestRegionReplicaReplication
provides a very uniform, unrealistic load, using different data could be
interesting. If someone would find the time to run a few tests or propose
some scenarios I would be grateful.
- I was thinking of providing a cleaner flush logic on replication side,
but my proposal might have too much overhead and the current logic while
having issues works after the previous fixes. The proposal can be found in
HBASE-23591, any feedback would be welcomed.

Thoughts?

[jira] [Created] (HBASE-23601) OutputSink.WriterThread exception gets stuck and repeated indefinietly

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23601:
---

 Summary: OutputSink.WriterThread exception gets stuck and repeated 
indefinietly
 Key: HBASE-23601
 URL: https://issues.apache.org/jira/browse/HBASE-23601
 Project: HBase
  Issue Type: Bug
  Components: read replicas
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros
 Fix For: 2.2.2


When a WriterThread runs into an exception (ie: NotServingRegionException), the 
exception is stored in the controller. It is never removed and can not be 
overwritten either.

 
{code:java}
public void run()  {
  try {
doRun();
  } catch (Throwable t) {
LOG.error("Exiting thread", t);
controller.writerThreadError(t);
  }
}{code}
Thanks to this every time PipelineController.checkForErrors() is called the 
same old exception is rethrown.

 

For example in RegionReplicaReplicationEndpoint.replicate there is a while loop 
that does the actual replicating. Every time it loops, it calls 
checkForErrors(), catches the rethrown exception, logs it but does nothing 
about it. This results in ~2GB log files in ~5min in my experience.

 

My proposal would be to clean up the stored exception when it reaches 
RegionReplicaReplicationEndpoint.replicate and make sure we restart the 
WriterThread that died throwing it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23591) Negative memStoreSizing

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23591:
---

 Summary: Negative memStoreSizing
 Key: HBASE-23591
 URL: https://issues.apache.org/jira/browse/HBASE-23591
 Project: HBase
  Issue Type: Bug
  Components: read replicas
Reporter: Szabolcs Bukros
 Fix For: 2.2.2


After a flush on the replica region the memStoreSizing becomes negative:

 
{code:java}
2019-12-17 08:31:59,983 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
0beaae111b0f6e98bfde31ba35be5408 : Replaying flush marker action: COMMIT_FLUSH 
table_name: "IntegrationTestRegionReplicaReplicati
on" encoded_region_name: "544affde3e027454f67c8ea46c8f69ee" 
flush_sequence_number: 41392 store_flushes { family_name: "f1" store_home_dir: 
"f1" flush_output: "3c48a23eac784a348a18e10e337d80a2" } store_flushes { 
family_name: "f2" store_home_dir: "f2" flush_output: 
"9a5283ec95694667b4ead2398af5f01e" } store_flushes { family_name: "f3" 
store_home_dir: "f3" flush_output: "e6f25e6b0eca4d22af15d0626d0f8759" } 
region_name: 
"IntegrationTestRegionReplicaReplication,,1576599911697.544affde3e027454f67c8ea46c8f69ee."
2019-12-17 08:31:59,984 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
0beaae111b0f6e98bfde31ba35be5408 : Received a flush commit marker with 
seqId:41392 and a previous prepared snapshot was found
2019-12-17 08:31:59,993 INFO org.apache.hadoop.hbase.regionserver.HStore: 
Region: 0beaae111b0f6e98bfde31ba35be5408 added 
hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f1/3c48a23eac784a348a18e10e337d80a2,
 entries=32445, sequenceid=41392, filesize=27.6 M
2019-12-17 08:32:00,016 INFO org.apache.hadoop.hbase.regionserver.HStore: 
Region: 0beaae111b0f6e98bfde31ba35be5408 added 
hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f2/9a5283ec95694667b4ead2398af5f01e,
 entries=12264, sequenceid=41392, filesize=10.9 M
2019-12-17 08:32:00,121 INFO org.apache.hadoop.hbase.regionserver.HStore: 
Region: 0beaae111b0f6e98bfde31ba35be5408 added 
hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/544affde3e027454f67c8ea46c8f69ee/f3/e6f25e6b0eca4d22af15d0626d0f8759,
 entries=32379, sequenceid=41392, filesize=27.5 M
2019-12-17 08:32:00,122 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
CustomLog decrMemStoreSize. Current: dataSize=135810071, getHeapSize=148400960, 
getOffHeapSize=0, getCellsCount=167243 delta: dataSizeDelta=155923644, 
heapSizeDelta=170112320, offHeapSizeDelta=0, cellsCountDelta=188399
2019-12-17 08:32:00,122 ERROR org.apache.hadoop.hbase.regionserver.HRegion: 
Asked to modify this region's 
(IntegrationTestRegionReplicaReplication,,1576599911697_0001.0beaae111b0f6e98bfde31ba35be54
08.) memStoreSizing to a negative value which is incorrect. Current 
memStoreSizing=135810071, delta=-155923644
java.lang.Exception
at 
org.apache.hadoop.hbase.regionserver.HRegion.checkNegativeMemStoreDataSize(HRegion.java:1323)
at 
org.apache.hadoop.hbase.regionserver.HRegion.decrMemStoreSize(HRegion.java:1316)
at 
org.apache.hadoop.hbase.regionserver.HRegion.decrMemStoreSize(HRegion.java:1303)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:5194)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:5025)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:1143)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:2232)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29754)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)

{code}
 

 

I added some custom logging to the snapshot logic to be able to see snapshot 
sizes:

 
{code:java}
2019-12-17 08:31:56,900 DEBUG org.apache.hadoop.hbase.regionserver.HRegion: 
0beaae111b0f6e98bfde31ba35be5408 : Replaying flush marker action: START_FLUSH 
table_name: "IntegrationTestRegionReplicaReplication" encoded_region_name: 
"544affde3e027454f67c8ea46c8f69ee" flush_sequence_number: 41392 store_flushes { 
family_name: "f1" store_home_dir: "f1" } store_flushes { family_name: "f2" 
store_home_dir: "f2" } store_flushes { family_name: "f3" store_home_dir: "f3" } 
region_name: 
"IntegrationTestR

[jira] [Created] (HBASE-23589) FlushDescriptor contains non-matching family/output combinations

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23589:
---

 Summary: FlushDescriptor contains non-matching family/output 
combinations
 Key: HBASE-23589
 URL: https://issues.apache.org/jira/browse/HBASE-23589
 Project: HBase
  Issue Type: Bug
  Components: read replicas
Affects Versions: 2.2.2
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


Flushing the active region creates the following files:
{code:java}
2019-12-13 08:00:20,866 INFO org.apache.hadoop.hbase.regionserver.HStore: Added 
hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f2/dab4d1cc01e44773bad7bdb5d2e33b6c,
 entries=49128, sequenceid
=70688, filesize=41.4 M
2019-12-13 08:00:20,897 INFO org.apache.hadoop.hbase.regionserver.HStore: Added 
hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f3/ecc50f33085042f7bd2397253b896a3a,
 entries=5, sequenceid
=70688, filesize=42.3 M
{code}
On the read replica region when we try to replay the flush we see the following:
{code:java}
2019-12-13 08:00:21,279 WARN org.apache.hadoop.hbase.regionserver.HRegion: 
bfa9cdb0ab13d60b389df6621ab316d1 : At least one of the store files in flush: 
action: COMMIT_FLUSH table_name: "IntegrationTestRegionReplicaReplication" 
encoded_region_name: "20af2eb8929408f26d0b3b81e6b86d47" flush_sequence_number: 
70688 store_flushes { family_name: "f2" store_home_dir: "f2" flush_output: 
"ecc50f33085042f7bd2397253b896a3a" } store_flushes { family_name: "f3" 
store_home_dir: "f3" flush_output: "dab4d1cc01e44773bad7bdb5d2e33b6c" } 
region_name: 
"IntegrationTestRegionReplicaReplication,,1576252065847.20af2eb8929408f26d0b3b81e6b86d47."
 doesn't exist any more. Skip loading the file(s)
java.io.FileNotFoundException: HFileLink 
locations=[hdfs://replica-1:8020/hbase/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f2/ecc50f33085042f7bd2397253b896a3a,
 
hdfs://replica-1:8020/hbase/.tmp/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f2/ecc50f33085042f7bd2397253b896a3a,
 
hdfs://replica-1:8020/hbase/mobdir/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f2/ecc50f33085042f7bd2397253b896a3a,
 
hdfs://replica-1:8020/hbase/archive/data/default/IntegrationTestRegionReplicaReplication/20af2eb8929408f26d0b3b81e6b86d47/f2/ecc50f33085042f7bd2397253b896a3a]
at org.apache.hadoop.hbase.io.FileLink.getFileStatus(FileLink.java:415)
at 
org.apache.hadoop.hbase.util.ServerRegionReplicaUtil.getStoreFileInfo(ServerRegionReplicaUtil.java:135)
at 
org.apache.hadoop.hbase.regionserver.HRegionFileSystem.getStoreFileInfo(HRegionFileSystem.java:311)
at 
org.apache.hadoop.hbase.regionserver.HStore$StoreFlusherImpl.replayFlush(HStore.java:2414)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayFlushInStores(HRegion.java:5310)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushCommitMarker(HRegion.java:5184)
at 
org.apache.hadoop.hbase.regionserver.HRegion.replayWALFlushMarker(HRegion.java:5018)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.doReplayBatchOp(RSRpcServices.java:1143)
at 
org.apache.hadoop.hbase.regionserver.RSRpcServices.replay(RSRpcServices.java:2229)
at 
org.apache.hadoop.hbase.shaded.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:29754)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:413)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:133)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:338)
at 
org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:318)
{code}
As you can see the flush_outputs are mixed up.

 

The issue is caused by HRegion.internalFlushCacheAndCommit. The code assumes 
"{color:#808080}stores.values() and storeFlushCtxs have same order{color}" 
which no longer seems to be true.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23566) Fix package/packet terminology problem in chaos monkeys

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23566:
---

 Summary: Fix package/packet terminology problem in chaos monkeys
 Key: HBASE-23566
 URL: https://issues.apache.org/jira/browse/HBASE-23566
 Project: HBase
  Issue Type: Improvement
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


There is a terminology problem in some of the network issue related chaos 
monkey actions. The universally understood technical term for network packet is 
packet, not "package".



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23352) Allow chaos monkeys to access cmd line params, and improve FillDiskCommandAction

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23352:
---

 Summary: Allow chaos monkeys to access cmd line params, and 
improve FillDiskCommandAction
 Key: HBASE-23352
 URL: https://issues.apache.org/jira/browse/HBASE-23352
 Project: HBase
  Issue Type: Improvement
  Components: integration tests
Affects Versions: 2.2.2
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


When integration tests are run through hbase cli the properties passed as cmd 
line params does not reach the chaos monkies. It is possible to define a 
property file, but it would be more flexible if we could also pick up 
properties from the command line.

Also I would like to improve FillDiskCommandAction, to stop the remote process 
if the call times out before it could have finished or was run without a size 
parameter.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Reopened] (HBASE-23085) Network and Data related Actions

[Szabolcs Bukros (Jira)]

 [ 
https://issues.apache.org/jira/browse/HBASE-23085?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Szabolcs Bukros reopened HBASE-23085:
-

Backport commit to branch-2 and branch-2.2

> Network and Data related Actions
> 
>
> Key: HBASE-23085
> URL: https://issues.apache.org/jira/browse/HBASE-23085
> Project: HBase
>  Issue Type: Sub-task
>  Components: integration tests
>        Reporter: Szabolcs Bukros
>    Assignee: Szabolcs Bukros
>Priority: Minor
> Fix For: 3.0.0
>
>
> Add additional actions to:
>  * manipulate network packages with tc (reorder, loose,...)
>  * add CPU load
>  * fill the disk
>  * corrupt or delete regionserver data files
> Create new monkey factories for the new actions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-23085) Network and Data related Actions

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-23085:
---

 Summary: Network and Data related Actions
 Key: HBASE-23085
 URL: https://issues.apache.org/jira/browse/HBASE-23085
 Project: HBase
  Issue Type: Sub-task
  Components: integration tests
Reporter: Szabolcs Bukros
Assignee: Szabolcs Bukros


Add additional actions to:
 * manipulate network packages with tc (reorder, loose,...)
 * add CPU load
 * fill the disk
 * corrupt or delete regionserver data files

Create new monkey factories for the new actions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (HBASE-22982) Send SIGSTOP to hang or SIGCONT to resume rs and add graceful rolling restart

[Szabolcs Bukros (Jira)]

Szabolcs Bukros created HBASE-22982:
---

 Summary: Send SIGSTOP to hang or SIGCONT to resume rs and add 
graceful rolling restart
 Key: HBASE-22982
 URL: https://issues.apache.org/jira/browse/HBASE-22982
 Project: HBase
  Issue Type: Sub-task
  Components: integration tests
Affects Versions: 3.0.0
Reporter: Szabolcs Bukros


* Add a Chaos Monkey action that uses SIGSTOP and SIGCONT to hang and resume a 
ratio of region servers.
 * Add a Chaos Monkey action to simulate a rolling restart including 
graceful_stop like functionality that unloads the regions from the server 
before a restart and then places it under load again afterwards.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)