[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013312#comment-17013312 ] Duo Zhang commented on HBASE-23326: --- I think you'd better change it to a small value first to see if it works? BTW, there is a UT to confirm this... > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013303#comment-17013303 ] Michael Stack commented on HBASE-23326: --- Ok. 7 days default. Will keep an eye on it. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013300#comment-17013300 ] Duo Zhang commented on HBASE-23326: --- See TimeToLiveProcedureWALCleaner. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013284#comment-17013284 ] Duo Zhang commented on HBASE-23326: --- Proc wal has its own TTL setting, which is much longer than normal WAL files. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013280#comment-17013280 ] Michael Stack commented on HBASE-23326: --- [~zhangduo] Made a sub-issue of this. Thanks. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013276#comment-17013276 ] Duo Zhang commented on HBASE-23326: --- This is a missing part. Let’s open a follow on issue. I think we can just follow what we have done for archiving WAL, that move the archived hfiles to the global archive directory after compaction. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013270#comment-17013270 ] Michael Stack commented on HBASE-23326: --- And this seem to be hanging around... %2C16000%2C1578524235076.1578524242389$masterproc$ though straight WALs are cleared. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17013235#comment-17013235 ] Michael Stack commented on HBASE-23326: --- [~zhangduo] I see that the regionprocedurestore WAL files are being moved to the /hbase/archive but I also see build up at /hbase/MasterProcs/archive of old procedure hfiles. I have 1500 files in it currently (running about 24 hours under load). Poking in PR I don't see obvious handling of this Region archive dir. What am I missing? Thanks. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003410#comment-17003410
]
Hudson commented on HBASE-23326:
Results for branch branch-2
[build #2396 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2396/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2396//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2396//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/2396//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17003283#comment-17003283
]
Hudson commented on HBASE-23326:
Results for branch master
[build #1578 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/1578/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1578//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1578//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/1578//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17000512#comment-17000512 ] Michael Stack commented on HBASE-23326: --- bq. This is exactly the thing I want to avoid... I do not think there are other data which needs to store like this way, please use system tables? You answer my suggestion using a condition I explicitly exclude. I don't feel strongly enough about this and what I suggest is too minor to spend cycles on. I'm just glad of the new storage implementation. Let me look at latest patch iteration. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998370#comment-16998370
]
Sean Busbey commented on HBASE-23326:
-
{quote}
bq. I'll try to get further comments into the context of the doc, but we
absolutely must have archiving for this to go into a branches-2. I've had too
many issues with things that thought they could skip archiving in the last 6
months already.
What's the problem? Need get the WAL back for recovery or debugging?
{quote}
yes, exactly. when something goes wrong there are already well-established ways
for me to get folks to tell hbase to leave stuff alone that's in the archiver
and for things to get cleaned up when we're done.
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16998146#comment-16998146
]
Duo Zhang commented on HBASE-23326:
---
{quote}
If master later wants to use filesystem for data that doesn't fit system tables
perhaps because it needed at startup, this would be natural place for it to put
its data beside the local region and WALs.
{quote}
This is exactly the thing I want to avoid... I do not think there are other
data which needs to store like this way, please use system tables?
{quote}
My suggestion is s/MasterProcs/master/. A dir named MasterProcs with a region
and WALs in it is kinda wonky
{quote}
In general, name it as MasterProcs means it contains the master procedure data,
you are free to store them in any format and now, the suitable way is to store
them with the region format. At the first place I even wanted to reuse the old
directory but then I found out that the name is 'MasterProcWALs', which is not
correct as now not only WALs are stored in it, so I named it 'MasterProcs'.
And I need to say again, in my mind there are no other data have to be stored
in this way, so I do not think we should make it general. Correct me if I'm
wrong.
{quote}
Suggestion is just get the WALs into existing archive and use its existing
archiving system (rather than have one for data WALs and another for procedure
WALs). We can write another if we have to because someone wants Master to carry
regions.
{quote}
In general I think this is fine. But after checking the code carefully, this is
not easy in the currect code base. The wal directory and archive directory must
be under the same wal root directory. And now since we changed the wal root
directory to /hbase/MasterProcs, there is no way for us to set the archive
directory to /hbase/oldWALs(unless we set the directory to ../oldWALs but I do
no think this is the correct way, it is like a CVE...). Another way is to write
the WAL to the normal /hbase/WALs directory and then we could use the
/hbase/oldWALs to archive the WAL files but this breaks my assumption above
that all the procedure data should be stored in the 'MasterProcs' so I do not
think this is the correct way.
Anyway, let me see if I can reuse the log cleaner code here.
Thanks.
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997902#comment-16997902 ] Michael Stack commented on HBASE-23326: --- bq. Do you have any ideas which data will be stored in the 'local' HRegion, besides procedure? I think we are talking past each other. My suggestion is s/MasterProcs/master/. A dir named MasterProcs with a region and WALs in it is kinda wonky. If it was named 'master', you'd know the master was the writer. If master later wants to use filesystem for data that doesn't fit system tables perhaps because it needed at startup, this would be natural place for it to put its data beside the local region and WALs. bq. I think this can be done in the future, but we need to refactor HRegion first if we want to do this. Ok. bq. ...master carrying table... This is not going to happen any time soon if at all. It had a bunch of time spent on it and it couldn't be made work. Will need a bunch of other time spent redoing startup. I think there other priorities that will come before this effort. Suggestion is just get the WALs into existing archive and use its existing archiving system (rather than have one for data WALs and another for procedure WALs). We can write another if we have to because someone wants Master to carry regions. bq. You can see the code in ProcedureTree for more details. Sounds good. Good stuff. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997897#comment-16997897
]
Duo Zhang commented on HBASE-23326:
---
{quote}
I'll try to get further comments into the context of the doc, but we absolutely
must have archiving for this to go into a branches-2. I've had too many issues
with things that thought they could skip archiving in the last 6 months already.
{quote}
What's the problem? Need get the WAL back for recovery or debugging?
{quote}
Sure. Was suggesting you use a more generic name, do should Master need to
persist other info, we could group it under a master dir
(s/MasterProcs/master/).
{quote}
This is what I want to avoid... Just store the data in system tables. Why we
have to use a local region is because that hbase:meta has not been intialized
when we loading procedures, so it should not use hbase:meta or other system
tables to store the data. It is fine to change the namespace to master, no
problem, but I still do not want to make it generic. Do you have any ideas
which data will be stored in the 'local' HRegion, besides procedure?
{quote}
RSS is an Interface. Was suggesting you pass an impl that only did the subset
needed to run local Region. I suggest this so local Region looks like any other
Region in the system.
{quote}
This does not work. Usually we just test whether the RegionServerServices is
null and then do something, so a non-null RegionServerServices actually means
we will enable lots of features. I think this can be done in the future, but we
need to refactor HRegion first if we want to do this.
{quote}
The WALs are differently named?
IIRC, just moving the WALs into the WALs dir gets them considered by the
general archiving system; no extra work to be done beyond the move.
{quote}
They are in the same name pattern so it will be a problem if master carrying
table, as it may overwrite the WAL file for master. Since we want to isolate
the procedure store data, let's implement the archive logic by our own, it is
easy and can use different configs comparing to normal wals.
{quote}
I suppose if out-of-order, once read into member, the procedures will be
ordered... .and we won't have odd stuff like seeing a subprocedure before its
parent (shouldn't matter).
{quote}
No, no problem. We will load all the procedures to memory first, and then build
the ProcedureTree, and then pass the procedures to the ProcedureLoader. You can
see the code in ProcedureTree for more details.
{quote}
Would be sweet if we could just move the WALs into recovered.edits dir (it is
named something else as flag to trigger different behavior? OK).
{quote}
Could be done in the future when we refactor HRegion? This needs a more high
level design to unify the different scenarios.
Thanks.
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Critical
> Fix For: 3.0.0, 2.3.0
>
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997870#comment-16997870 ] Michael Stack commented on HBASE-23326: --- bq. ...so let's isolate it from the normal region store. Sure. Was suggesting you use a more generic name, do should Master need to persist other info, we could group it under a master dir (s/MasterProcs/master/). bq. if we pass a RegionServerServices in... RSS is an Interface. Was suggesting you pass an impl that only did the subset needed to run local Region. I suggest this so local Region looks like any other Region in the system. Yeah, HRegion needs refactoring. Ideally it'd be moved out into its own module. Probably have to do hbase-wal module first. bq. if we enable region on master The WALs are differently named? IIRC, just moving the WALs into the WALs dir gets them considered by the general archiving system; no extra work to be done beyond the move. bq. ...order is not important I suppose if out-of-order, once read into member, the procedures will be ordered... .and we won't have odd stuff like seeing a subprocedure before its parent (shouldn't matter). Would be sweet if we could just move the WALs into recovered.edits dir (it is named something else as flag to trigger different behavior? OK). > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997746#comment-16997746 ] Sean Busbey commented on HBASE-23326: - I'll try to get further comments into the context of the doc, but we absolutely must have archiving for this to go into a branches-2. I've had too many issues with things that thought they could skip archiving in the last 6 months already. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Critical > Fix For: 3.0.0, 2.3.0 > > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997737#comment-16997737
]
Duo Zhang commented on HBASE-23326:
---
{quote}
Thanks for putting up the doc. Can we have 'comment' access. Here are a few
notes in meantime (some carried over from github comments):
{quote}
Done.
{quote}
On layout, could have a 'master' namespace so master Region is in same place in
filesystem (Might be too awkward excluding this namespace from consideration in
general processing).
Or, here you make a MasterProcs directory. Old system had a MasterProcWALs dir.
Make instead a generic 'master' dir at top-level into which we put all stuff
master wants to persist to filesystem of which these new procedures WALs would
be first.
{quote}
The design here is to make the procedure store be self-managed. IMO, most data
should be stored in hbase:meta, or other system tables. And why we need this
special store is that, inializing and assigning meta depend on the procedure
framework so we can not use hbase:meta store these things. This should not be a
common case, so let's isolate it from the normal region store.
{quote}
You cannot pass a RegionServerServices that has the special implementations of
flush/compaction/rolling? Just to minimize how this Region implementation
deviates from the norm.
{quote}
It is a 'local' HRegion, so in general, it should not have a
RegionServerServices along with it. And in fact, if we pass a
RegionServerServices in, lot's of other features will be activated, such as
quota, metrics, etc. This will cause problem as if we do not enable table on
master, some of the components are not initialized. Of course, metrics is
useful here, but can be a follow on.
And in general, I think we can do some refactoring on HRegion, to make it
decouple with RegionServerServices, and for the optional features, we can add
some interface to make them pluggable, then the code here will be more clean.
But anyway, as said above, this should not be a common case, so do not need to
be hurry on the refactoring.
{quote}
WAL dirs will be deleted/cleanedup after WALs are moved to recovered.edits?
There'll be no accumulation of WALs? What about archiving? Peter figured how to
get the MasterProcWALs into the general WAL archive. Maybe no archiving of
these WALs?
{quote}
I do not think we can archive the WALs to the general place as if we enable
region on master, it will mess things up. Now I haven't take care of this part
yet, the intention is to just delete the WALs in the first version. And later,
we could implement our own archiving logic, it is easy I think. Anyway, the
design here is to be self-managed. And for tracing the problem, if we assume
that the HRegion and WAL framework are fine(If it is not fine, then we should
find out on the normal read/write path), then the problem should be in our code
which read/write to the HRegion. So maybe we could enable multi version and
keep deleted cells on this region, to make it more debug friendly.
{quote}
For recovered.edits, they content is supposed to be 'sorted'. When we move WALs
to recovered.edits, they will be 'sorted' because we write in procedure order?
Is there anything we need to do to ensure edits go into the WAL 'ordered'?
{quote}
Technically, they do not need to be 'sorted'. As there are sequece ids in the
WALEntry and we do not do compaction then replaying, order is not important.
And why we make them sorted is because performance. As when splitting we can
know all the sequence ids of the WALEntries contained in a splitted WAL file,
so we can just name it with the sequence ids. Then when replaying, we can
quickly filter out the unnecessary WAL files. But here, since we do not need to
split, it is not necessary to read the files again and rename them...
And that's why I use a different name of the directory to put these WAL files.
You can see the modification in HRegion, I added a special config to specify
the special directory to place recovered'edits. If this option is set, then the
logic is a bit different, where we will not filter out any WAL files, and do
not check its name parttern.
Thanks.
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overload
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16997496#comment-16997496 ] Michael Stack commented on HBASE-23326: --- Thanks for putting up the doc. Can we have 'comment' access. Here are a few notes in meantime (some carried over from github comments): On layout, could have a 'master' namespace so master Region is in same place in filesystem (Might be too awkward excluding this namespace from consideration in general processing). Or, here you make a MasterProcs directory. Old system had a MasterProcWALs dir. Make instead a generic 'master' dir at top-level into which we put all stuff master wants to persist to filesystem of which these new procedures WALs would be first. You cannot pass a RegionServerServices that has the special implementations of flush/compaction/rolling? Just to minimize how this Region implementation deviates from the norm. WAL dirs will be deleted/cleanedup after WALs are moved to recovered.edits? There'll be no accumulation of WALs? What about archiving? Peter figured how to get the MasterProcWALs into the general WAL archive. Maybe no archiving of these WALs? For recovered.edits, they content is supposed to be 'sorted'. When we move WALs to recovered.edits, they will be 'sorted' because we write in procedure order? Is there anything we need to do to ensure edits go into the WAL 'ordered'? Nice writeup. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996953#comment-16996953 ] Duo Zhang commented on HBASE-23326: --- The ongoing design doc. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[
https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996951#comment-16996951
]
Duo Zhang commented on HBASE-23326:
---
{quote}
Is it this region co-located with Master?
{quote}
Yes, you can consider it as a 'local' HRegion. You can not access it through
the network.
> Implement a ProcedureStore which stores procedures in a HRegion
> ---
>
> Key: HBASE-23326
> URL: https://issues.apache.org/jira/browse/HBASE-23326
> Project: HBase
> Issue Type: Improvement
> Components: proc-v2
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> So we can resue the code in HRegion for persisting the procedures, and also
> the optimized WAL implementation for better performance.
> This requires we merge the hbase-procedure module to hbase-server, which is
> an anti-pattern as we make the hbase-server module more overloaded. But I
> think later we can first try to move the WAL stuff out.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996935#comment-16996935 ] Allan Yang commented on HBASE-23326: It's a nice approach to put all the procedures in a region. Is it this region co-located with Master? If so, it is great! Indeed a design doc is needed to understand it. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996917#comment-16996917 ] Duo Zhang commented on HBASE-23326: --- Let me write a simple design doc. The basic idea is simple but maybe I should explain the internal a bit. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (HBASE-23326) Implement a ProcedureStore which stores procedures in a HRegion
[ https://issues.apache.org/jira/browse/HBASE-23326?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16996766#comment-16996766 ] Sean Busbey commented on HBASE-23326: - Can we get some kind of design doc or overview doc of goals and approach? I think one of the big tripping up points for the existing procedure store is that folks have to start at the code without any real guideposts. > Implement a ProcedureStore which stores procedures in a HRegion > --- > > Key: HBASE-23326 > URL: https://issues.apache.org/jira/browse/HBASE-23326 > Project: HBase > Issue Type: Improvement > Components: proc-v2 >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > So we can resue the code in HRegion for persisting the procedures, and also > the optimized WAL implementation for better performance. > This requires we merge the hbase-procedure module to hbase-server, which is > an anti-pattern as we make the hbase-server module more overloaded. But I > think later we can first try to move the WAL stuff out. -- This message was sent by Atlassian Jira (v8.3.4#803005)
