[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582371#comment-17582371
]
Hudson commented on HBASE-26969:
Results for branch branch-2.5
[build #190 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/190/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/190/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/190/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/190/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/190/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17582129#comment-17582129
]
Hudson commented on HBASE-26969:
Results for branch branch-2
[build #621 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/621/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/621/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/621/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/621/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2/621/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17581779#comment-17581779
]
Peter Somogyi commented on HBASE-26969:
---
{quote}HBASE-25970{color:#172b4d} is another dependency I missed in
{color}HBASE-27154{color:#172b4d}. To shorten the review cycles instead of
backporting it separately I included the changes it contained in this PR.
Please let me know if that would not be acceptable and I prepare a separate PR
for that backport.{color}
{quote}
I've split this up into 2 commits so it will be easier to track in Jira.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 2.5.1, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17580779#comment-17580779
]
Szabolcs Bukros commented on HBASE-26969:
-
[~apurtell] [~huaxiangsun] Please find #4712 with the branch-2 backport.
Please note I made some additional changes:
* [#4617|https://github.com/apache/hbase/pull/4617] left 2 test classes in
the code that the original commit deleted and I had to delete them here.
* HBASE-25970 is another dependency I missed in HBASE-27154. To shorten the
review cycles instead of backporting it separately I included the changes it
contained in this PR. Please let me know if that would not be acceptable and I
prepare a separate PR for that backport.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17579144#comment-17579144
]
Huaxiang Sun commented on HBASE-26969:
--
We are planing to spin 2.5.0 RC next Monday, [~bszabolcs] , do you think it can
be merged by then? Otherwise, it will push out o 2.5.1.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565961#comment-17565961
]
Andrew Kyle Purtell commented on HBASE-26969:
-
Happy to help on HBASE-27154 [~bszabolcs].
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565318#comment-17565318
]
Szabolcs Bukros commented on HBASE-26969:
-
[~apurtell] branch-2 misses some very important MOB related commits that this
change relies on and we have to backport those first. It's tracked in
HBASE-27154. The first and most complex backport is done, the rest should be
easier but I have deadlines coming up and will not be able to continue this for
a few weeks. If you could do the rest or find someone to do the rest of the
missing commits, I would be happy to prepare a backport for this commit though.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17565203#comment-17565203
]
Andrew Kyle Purtell commented on HBASE-26969:
-
This can still make 2.5 if a branch-2 patch lands in the next couple of days.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17557623#comment-17557623
]
Hudson commented on HBASE-26969:
Results for branch master
[build #617 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/617/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/617/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/617/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/master/617/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17556751#comment-17556751
]
Wellington Chevreuil commented on HBASE-26969:
--
Thanks for your patience in answering/addressing all my questions here,
[~bszabolcs]. I had now merged this PR into master branch. Tried to cherry-pick
into branch-2, bug got a lot of conflicts. Mind push another PR for branch-2?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-4
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544550#comment-17544550
]
Szabolcs Bukros commented on HBASE-26969:
-
{quote}
So p1 and p2 can have references to p even when p is no longer in meta?
{quote}
Not exactly. In this scenario p no longer exists in our outside the meta. It is
not referenced either. The mobfile "_p" exists and is
referenced from p1 and p2. It was created by p but it exists outside the data
folder in an entirely different structure under /hbase/mobdir. It is not and
never was part of p. It was only referenced from p.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17544492#comment-17544492
]
Wellington Chevreuil commented on HBASE-26969:
--
{quote}
We have 2 new regions p1, p2 and p is archived, maybe even deleted altogether.
Both p1 and p2 are online and contain references to "_p"
mobfile but we have no way of knowing we should search these regions for
references. So the master cleaner have to read every single hfile to find the
references in p1 and p2.
{quote}
So p1 and p2 can have references to p even when p is no longer in meta? If
that's the case, then yeah, we need master cleaner to check everything.
I was wondering if p would be kept in meta as split until p1 and p2 compacted
and had no reference to files from p. And similarly for merges, then we could
just scan meta for regions in these states.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542548#comment-17542548
]
Szabolcs Bukros commented on HBASE-26969:
-
> the master cleaner could check only files from regions not online on any RS
That would not be enough. Consider the following scenario. Region p creates a
mobfile with a name of "_p". While region p is online the rs
cleaner can identify that "_p" belongs to this region and can
clean it up if it is no longer referenced from said region. Now let's split
region p. We have 2 new regions p1, p2 and p is archived, maybe even deleted
altogether. Both p1 and p2 are online and contain references to
"_p" mobfile but we have no way of knowing we should search
these regions for references. So the master cleaner have to read every single
hfile to find the references in p1 and p2.
The mobfiles keep their name until a major compaction runs.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17542067#comment-17542067
]
Wellington Chevreuil commented on HBASE-26969:
--
{quote}
The problem is that when we can not easily identify which region should contain
references to a given MOB file we have absolutely no way to tell and have to
read every single hfile's metadata to check for references.
{quote}
I get that, my question here is if there is a way the master cleaner could
check only files from regions not online on any RS (as online regions would
already be checked by the cleaner running on each RS). If we can trust meta
listing of split/merged regions, then we could only iterate through those,
instead of every single region dir under the root dir. But if split/merged
region dirs are still kept under root dir even after those regions are removed
from meta, then we can't rely on meta.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541025#comment-17541025
]
Szabolcs Bukros commented on HBASE-26969:
-
[~zhangduo] I'm sorry, I might have misspoken. We do not need an extra SFT.
Having the references in hfile metadata is sufficient. It's just slow and
clunky. But it works. I hoped a better way of storing this data could be found,
but as you have pointed out that is not necessary.
SFT is only linked to this issue, because SFT and removing renames are
thematically connected, I'm relying on some tools/solutions added to support
SFT and removing renames makes things more complicated so instead of changing
the default behavior the idea was to only remove them when SFT, that removed
the other renames, is enabled.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541006#comment-17541006
]
Duo Zhang commented on HBASE-26969:
---
{noformat}
That is true. We do not use SFT. But based on similar changes the renames would
be only eliminated if SFT is enabled, not by default. Also it relies on
WriterCreationTracker which is mostly an SFT tool.
{noformat}
As long as we have all the file references in the normal store file's metadata,
why we still need an extra SFT?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17541004#comment-17541004
]
Szabolcs Bukros commented on HBASE-26969:
-
[~wchevreuil]
> Or is it that we lingering region dirs even after the parent SPLIT/MERGE got
>removed from META?
Yes. these regions are clean up by a chore and linger after no longer used. But
this does not really matter. The problem is that when we can not easily
identify which region should contain references to a given MOB file we have
absolutely no way to tell and have to read every single hfile's metadata to
check for references.
[~zhangduo]
> we do not use SFT at all the MOB regions
That is true. We do not use SFT. But based on similar changes the renames would
be only eliminated if SFT is enabled, not by default. Also it relies on
WriterCreationTracker which is mostly an SFT tool.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538828#comment-17538828
]
Duo Zhang commented on HBASE-26969:
---
So I think the title of this PR should be changed than? This is not about SFT,
we do not use SFT at all the MOB regions. It is just 'Eliminate MOB renames'...
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17538825#comment-17538825
]
Wellington Chevreuil commented on HBASE-26969:
--
Double posting my comments from the PR. Functionally, the Master cleaner looks
correct to me. Thinking if there's a possible more efficient alternative, can
we rely on meta to fetch the parent SPLIT/MERGE regions? I guess those are the
ones that would not be online anywhere. Or is it that we lingering region dirs
even after the parent SPLIT/MERGE got removed from META?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17537787#comment-17537787
]
Szabolcs Bukros commented on HBASE-26969:
-
{{{quote}}}
I guess if we want to compact the mob files, we always need to compact the
normal files which references the mob files so we can update the references in
the metadata?
{quote}
Yes, that is how it works.
{{{quote}}}
The mob files should have a different name prefix or under a different
directory?
{{{}{quote}{}}}{{{}{}}}
They have a different directory structure.
"/mobdir/data/default/table_name/a0209c070c85d1e4d500af8ba33c3c02/cf"
They are stored fully separated. Please note these regions only contain mob
files and are fully independent from the referencing regions. A single mob
region could theoretically contain every MOB file in hbase regardless of where
it is referenced from. Also their naming convention is different. For us the
only important thing is that it ends with "_", so something like this:
"0cc175b9c0f1b6a831c399e2697726612022050314ecf20b51674cd6bd647bfb2d88b1ff_b593e96e821ba6211d8a4b101a88"
{quote}
So at least for loading, there will be no problem
{quote}
That's true. Reads are very straightforward.
{{{quote}}}
I think the only problem here is how do we clean up the half written mob files,
I think the logic is mainly the same with what we have now, get all the mob
refs from all the normal storefiles, to construct the base list, and then get
all the mob files which are currently being written, all MOB files besides them
are the ones should be deleted.
{quote}{{{}{}}}
{{That is part of the problme, yes. To have access to the half written mob file
list the cleaner have to run on the RS. But each RS only has access to it's own
}}half written mob file list so each can only clean a subset of the existing
mob files. To be precise if a mob file name ends with a region's name that is
hosted on the current RS then the cleaner can decide if it can be archived or
not. Unfortunately with merges and splits regions get archived so after a point
there will be mob files containing names of regions not hosted on any RS and
none of the cleaners running on RSes could clean these up. So we need one more
cleaner specifically for these (I put it on master to replace the original
cleaner), that have to read every available hfile to make sure we have every
active mob reference and are able to decide if a mob file created by a since
archived region can be archived or not.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536730#comment-17536730
]
Duo Zhang commented on HBASE-26969:
---
OK, so when flushing, we will write a mob file, and then added it to the
metadata of the normal file so when reading, we know where to find the files.
And in this way, I guess if we want to compact the mob files, we always need to
compact the normal files which references the mob files so we can update the
references in the metadata?
Then I do not think we need to use SFT to track the mob files.
The mob files should have a different name prefix or under a different
directory? Or at least, reading the header can let us know that it is a MOB
file? When loading, we just collect all the mob refs from all the normal
storefiles, then we know the list of mob files. So at least for loading, there
will be no problem as we will not load the broken MOB files if we write the
files directly to the final directory.
I think the only problem here is how do we clean up the half written mob files,
I think the logic is mainly the same with what we have now, get all the mob
refs from all the normal storefiles, to construct the base list, and then get
all the mob files which are currently being written, all MOB files besides them
are the ones should be deleted.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536589#comment-17536589
]
Szabolcs Bukros commented on HBASE-26969:
-
[~zhangduo] thank you for taking time and reading this!
As far as I understand when a storeFile has data that is stored in a mob file
then that storeFile's metadata will have a reference to these mob files. So
when a scan request tries to read the data it knows which mob file to check.
This is the only tracking we have.
{code:java}
storeFile.getMetadataValue(HStoreFile.MOB_FILE_REFS); {code}
For cleanup the chore have to know which mob files are currently actively
referenced. To get this lit, the chore check's the metadata of every single
storeFile hbase have in a mob enabled CF, and collects the references from
them. It just iterates through the /data folder table by table.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17536534#comment-17536534
]
Duo Zhang commented on HBASE-26969:
---
I'm not very familiar with the MOB related code, but how do we track and clean
MOB files before we have SFT? If it is similiar to the normal hfiles, then I
think SFT will also work. If it is not, then I think we should review again to
see whether SFT is the suitable way here.
So mind posting the original tracking way of MOB files first here?
Thanks.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Sub-task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17534360#comment-17534360
]
Szabolcs Bukros commented on HBASE-26969:
-
I would like to start by stating that this issue grow bigger than just removing
the renames and exposed multiple issues in the MOB-SFT interaction.
I have uploaded a draft PR containing my changes. I intend to use it as a
reference to show the issues when it comes to using MOB on FileBased SFT.
My main problem was that while MOB files were already tracked in the hfile
metadata, the "single source of truth" is widely distributed and not easily
available.
Both the WriterCreationTracker and the StoreFileTracker are RS based data and
the MOB cleaner needs it to work reliably when FileBased SFT is used. Exposing
this data and allowing the Master to request this from RSes, collect it and run
the cleaner based on this, while technically possible, looked less than
optimal. It would result in a single cluster wide spike that we should try to
avoid and considering the delay that certain RSes could have (uneven load, GC
pauses, etc) the data can be already outdated by the time the collection is
done. So instead I tried to move the cleaner to the RSes. This solution also
had it's drawbacks.
MOB file names contain the encoded name of the region that created them so the
RS hosting that specific region can check it's hfiles for references and can
clean it up if it does not find anything. The problem comes with merge/split
parent regions. When the parent region is archived the new region's hfiles will
still hold references to the old MOB files but now the only way to make sure if
the old MOB file is referenced or not is to check every single hfile in every
store belonging to the same columnfamily, because we can not tell based on it's
name where it could be referenced from. Like the old cleaner did. So while I
moved the MOB cleaner to the RS level and reduced it's scope to only clean up
MOB files belonging to regions hosted by that RS I had to leave a "global" MOB
cleaner running on Master to deal with MOB files created by archived regions
but potentially still being referenced. And I think this is very ugly.
This whole process could have been significantly simpler if we would have
tracker files in MOB stores but then we would have TWO competing sources of
truth. The tracker files and the hfile metadata.
HBASE-27017 is a related issue where the snapshot code tries to get the active
MOB files based on the configured SFT, but since MOB stores do not have tracker
files it returns an empty list. If the store had tracker files it would work.
Without a tracker file we either include every MOB files in the dir (garbage
included) or scan every single hfile metadata for MOB references.
What I'm trying to say is that while I think my solution would work and solve
the immediate issues I would much prefer if there would be a centralized,
easily available active MOB list and create a solution based on that.
[~apurtell] ,[~zhangduo],[~elserj] ,[~wchevreuil] What do you think?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533854#comment-17533854
]
Szabolcs Bukros commented on HBASE-26969:
-
Thanks [~apurtell] !
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533837#comment-17533837
]
Andrew Kyle Purtell commented on HBASE-26969:
-
[~bszabolcs] I added
bq. The file based StoreFileTracker, FileBasedStoreFileTracker, is currently
incompatible with the Medium Objects (MOB) feature. Do not enable them
together.
to the release notes of HBASE-26067 and HBASE-26826, so they will appear in all
relevant release notes. If you would prefer an alternative, please provide it
here and I will update the text again.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17533817#comment-17533817
]
Szabolcs Bukros commented on HBASE-26969:
-
[~apurtell] It looks like the MOB feature is currently incompatible with
FileBased SFT. Without this issue fixed the currently
written/temporary/outdated/trash files in the store dir can break the
MobFileCleanerChore and the related issue shows that snapshotting a MOB enabled
table while FileBased SFT is used results in dataloss. Since 2.5.0 being so
close to release this fact should be documented somewhere.
I'm planning to add a more detailed description of the issues I have
encountered while trying to make these features work together as soon as I can
publish a PR for reference.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531442#comment-17531442
]
Andrew Kyle Purtell commented on HBASE-26969:
-
(y)
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17531420#comment-17531420
]
Szabolcs Bukros commented on HBASE-26969:
-
[~apurtell] Please bump it.
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
> Fix For: 2.5.0, 2.6.0, 3.0.0-alpha-3
>
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
[jira] [Commented] (HBASE-26969) Eliminate MOB renames when SFT is enabled
[
https://issues.apache.org/jira/browse/HBASE-26969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17530983#comment-17530983
]
Andrew Kyle Purtell commented on HBASE-26969:
-
Going to roll 2.5.0RC0 soon. Keep this in for imminent resolution or bump it?
> Eliminate MOB renames when SFT is enabled
> -
>
> Key: HBASE-26969
> URL: https://issues.apache.org/jira/browse/HBASE-26969
> Project: HBase
> Issue Type: Task
> Components: mob
>Affects Versions: 2.5.0, 3.0.0-alpha-3
>Reporter: Szabolcs Bukros
>Assignee: Szabolcs Bukros
>Priority: Major
>
> MOB file compaction and flush still relies on renames even when SFT is
> enabled.
> My proposed changes are:
> * when requireWritingToTmpDirFirst is false during mob flush/compact instead
> of using the temp writer we should create a different writer using a
> {color:#00}StoreFileWriterCreationTracker that writes directly to the mob
> store folder{color}
> * {color:#00}these StoreFileWriterCreationTracker should be stored in
> the MobStore. This would requires us to extend MobStore with a createWriter
> and a finalizeWriter method to handle this{color}
> * {color:#00}refactor {color}MobFileCleanerChore to run on the RS
> instead on Master to allow access to the
> {color:#00}StoreFileWriterCreationTracker{color}s to make sure the
> currently written files are not cleaned up
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
