[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17512927#comment-17512927
]
Hudson commented on HBASE-26067:
Results for branch branch-2.5
[build #78 on
builds.a.o|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
(x) {color:red}-1 client integration test{color}
-- Something went wrong with this stage, [check relevant console
output|https://ci-hbase.apache.org/job/HBase%20Nightly/job/branch-2.5/78//console].
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468703#comment-17468703
]
Josh Elser commented on HBASE-26067:
{quote}Let me go for a merge and then resolve this issue
{quote}
Yeah, looks like this only affects the FILE tracker. Interesting. Thanks for
merging Duo! Will pick up more over in HBASE-26584.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 2.6.0, 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468665#comment-17468665 ] Duo Zhang commented on HBASE-26067: --- OK, so we are on the same page now :) Let me go for a merge and then resolve this issue, we can continue the discussion under HBASE-26584. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468655#comment-17468655
]
Josh Elser commented on HBASE-26067:
{quote}We can file an issue to fix the above problem first. I think if all the
UTs are OK, we can merge it first, and then start to fix other problems.
[|https://issues.apache.org/jira/secure/EditComment!default.jspa?id=13387830=17468318]{quote}
Let me test real quick to figure out if this also affects HBase when the
default tracker is being used. If the compaction failure only happens with the
FILE tracker, I agree we can move ahead with a merge :)
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468631#comment-17468631
]
Wellington Chevreuil commented on HBASE-26067:
--
{quote}
So for the follow on issues, we could apply the patches to both master and
branch-2. The PR for HBASE-26585 is already here...
{quote}
I'm ok with that. Regarding [~elserj] reported issue, is it introduced by the
SFT backport? Have you checked if this was not already happening at branch-2 on
its own, [~elserj] ?
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468624#comment-17468624 ] Duo Zhang commented on HBASE-26067: --- https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067-branch-2/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/ https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067-branch-2/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop2_29/ https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067-branch-2/7/General_20Nightly_20Build_20Report/ JDK8 nightly jobs are both fine, general report is also good. For me I think we could merge it back to branch-2 now. So for the follow on issues, we could apply the patches to both master and branch-2. The PR for HBASE-26585 is already here... WDYT? [~elserj] [~wchevreuil]. Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468318#comment-17468318 ] Duo Zhang commented on HBASE-26067: --- Thanks [~elserj]. We can file an issue to fix the above problem first. I think if all the UTs are OK, we can merge it first, and then start to fix other problems. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468268#comment-17468268
]
Josh Elser commented on HBASE-26067:
{quote}I'm looking to see if I can spot something obvious, but maybe the same
thing happened here.
{quote}
I think I see one thing in DefaultCompactor. I'll tag you on a PR
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468264#comment-17468264
]
Josh Elser commented on HBASE-26067:
Ran a simple {{hbase pe randomWrite}} and I'm noticing one thing that we also
saw when backporting this to branch-2.4 (internally)
{noformat}
2022-01-03 17:40:47,184 INFO [MemStoreFlusher.1]
regionserver.DefaultStoreFlusher(81): Flushed memstore data size=73.26 MB at
sequenceid=4537 (bloomFilter=true),
to=hdfs://mizar.cloudera:8020/hbase-2.6/data/default/TestTable/951d5e954f95ab58f224fe80a77bea56/info0/7732094
d69a94641a56965c4fb0d1947 {noformat}
and
{noformat}
2022-01-03 17:41:18,319 ERROR [regionserver/localhost:16020-shortCompactions-0]
regionserver.CompactSplit(670): Compaction failed
region=TestTable,0004054490,1641249249856.2dc7251c6eceb660b9c7bb0b587db913.,
storeName=2dc7251c6eceb660b9c7bb0b587db913/info0, priority=6,
startTime=1641249666161
java.io.IOException: Root-level entries already added in single-level mode
at
org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexWriter.writeSingleLevelIndex(HFileBlockIndex.java:1136)
at
org.apache.hadoop.hbase.io.hfile.CompoundBloomFilterWriter$MetaWriter.write(CompoundBloomFilterWriter.java:279)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl$1.writeToBlock(HFileWriterImpl.java:713)
at
org.apache.hadoop.hbase.io.hfile.HFileBlock$Writer.writeBlock(HFileBlock.java:1205)
at
org.apache.hadoop.hbase.io.hfile.HFileWriterImpl.close(HFileWriterImpl.java:660)
at
org.apache.hadoop.hbase.regionserver.StoreFileWriter.close(StoreFileWriter.java:377)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.commitWriter(DefaultCompactor.java:70)
at
org.apache.hadoop.hbase.regionserver.compactions.Compactor.compact(Compactor.java:386)
at
org.apache.hadoop.hbase.regionserver.compactions.DefaultCompactor.compact(DefaultCompactor.java:62)
at
org.apache.hadoop.hbase.regionserver.DefaultStoreEngine$DefaultCompactionContext.compact(DefaultStoreEngine.java:125)
at org.apache.hadoop.hbase.regionserver.HStore.compact(HStore.java:1141)
at org.apache.hadoop.hbase.regionserver.HRegion.compact(HRegion.java:2388)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.doCompaction(CompactSplit.java:654)
at
org.apache.hadoop.hbase.regionserver.CompactSplit$CompactionRunner.run(CompactSplit.java:697)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748) {noformat}
We had figured out that this started happening when we backported HBASE-26271.
[~bszabolcs] actually realized that we had transposed the arguments in
{{sinkFactory.createWriter}} (substituting the dropCache for
request.isMajor()). I'm looking to see if I can spot something obvious, but
maybe the same thing happened here.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17468198#comment-17468198 ] Josh Elser commented on HBASE-26067: Thanks for doing all of this lifting, Duo! Let me try out the branch locally. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467828#comment-17467828
]
Duo Zhang commented on HBASE-26067:
---
HBASE-26641 helps a bit.
This was the command I used for testing the branch
{noformat}
mvn test -fn -PrunAllTests -Dsurefire.rerunFailingTestsCount=2
-Dhadoop.profile=3.0 &>test.log BalancerTestBase.testWithCluster:561->BalancerTestBase.testWithCluster:594->BalancerTestBase.assertClusterAsBalanced:213
All servers should have load no more than 60.
server=srv1704715723,3637,2575641733119932236 , load=6
1
[ERROR] Run 2:
TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster:41->BalancerTestBase.testWithCluster:561->BalancerTestBase.testWithCluster:594->BalancerTestBase.assertClusterAsBalanced:215
All servers should have load no less than 60.
server=srv1613724309,9647,7201182604512302666 , load=5
9
[ERROR] Run 3:
TestStochasticLoadBalancerRegionReplicaLargeCluster.testRegionReplicasOnLargeCluster:41->BalancerTestBase.testWithCluster:561->BalancerTestBase.testWithCluster:594->BalancerTestBase.assertClusterAsBalanced:213
All servers should have load no more than 60.
server=srv805620918,32796,3201540874026950896 , load=6
1
[INFO]
[ERROR] Errors:
[ERROR] org.apache.hadoop.hbase.replication.TestReplicationDroppedTables.null
[ERROR] Run 1:
TestReplicationDroppedTables.testEditsDroppedWithDroppedTableNS:128->testEditsBehindDroppedTable:179->Object.wait:462->Object.wait:-2
» TestTimedOut
[ERROR] Run 2: TestReplicationDroppedTables » Appears to be stuck in thread
RS-EventLoopGrou...
{noformat}
For me I do not think they are related to our changes here, and
TestStochasticLoadBalancerRegionReplicaLargeCluster is on the flaky list of
branch-2 and TestReplicationDroppedTables is known to be flaky in our pre
commit.
So I think we are good to go. [~elserj] WDTY? Thanks.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
> Fix For: 3.0.0-alpha-3
>
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467639#comment-17467639 ] Duo Zhang commented on HBASE-26067: --- TestSplitTableRegionProcedure and TestReplicationMetricsforUI passed for me locally, so should just be flaky. Let's see the next nightly build. And I will run all the UTs locally too. Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467638#comment-17467638 ] Duo Zhang commented on HBASE-26067: --- OK, TestMasterFailoverWithProcedures fails 100% on branch-2, so it is not our fault. https://nightlies.apache.org/hbase/HBase/HBase-Find-Flaky-Tests/branch-2/498/output/dashboard.html Let me check TestSplitTableRegionProcedure and TestReplicationMetricsforUI. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467577#comment-17467577 ] Duo Zhang commented on HBASE-26067: --- OK, it turns out that the problem for TestMergesSplitsAddToTracker is a behavior change of Admin.split method. On master it will wait for split to finish while on branch-2 it will return immediately after sending the request to master. So move HBASE-26639 under HBASE-26584 instead of here as it is not blocker now. Let me check other failed UTs. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467558#comment-17467558 ] Duo Zhang commented on HBASE-26067: --- The failed UTs are TestMasterFailoverWithProcedures, TestMergesSplitsAddToTracker, TestSplitTableRegionProcedure, TestReplicationMetricsforUI. The jdk11 failure is expected, our build scripts will idenfity HBASE-26067-branch-2 as a feature branch which based on master, so it will not add -Dhadoop.profile=3.0 when executing maven so it will lead to a failure. Let me take a look at these UTs. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467449#comment-17467449 ] Duo Zhang commented on HBASE-26067: --- I've done a cherry-pick range to branch-2. https://github.com/apache/hbase/tree/HBASE-26067-branch-2 Let me set up a nightly build for this branch, I believe there will be a bunch of UT failures, but should not be too many problems, just need to fix some fundamental missing parts I think, for example, we still have namespace table on branch-2 and it will also be effect by SFT. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467416#comment-17467416 ] Duo Zhang commented on HBASE-26067: --- I could have a try too. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467252#comment-17467252 ] Josh Elser commented on HBASE-26067: [~zhangduo] I started fixing conflicts on the first commit but that was as far as I got before the holidays here in the US. If you have the cycles to work on it, that's fine by me. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17467103#comment-17467103 ] Duo Zhang commented on HBASE-26067: --- Any updates here [~elserj]? Do you need any help on the backporting? Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464230#comment-17464230 ] Duo Zhang commented on HBASE-26067: --- Thanks [~elserj] [~wchevreuil] and [~bszabolcs] for the great work! > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17464032#comment-17464032 ] Josh Elser commented on HBASE-26067: I've posted the vote result to the dev list and merged this set of changes into master. It looks like the backport to branch-2 will be a little bit of work. If I make meaningful progress, I'll update here with a branch > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > Fix For: 3.0.0-alpha-3 > > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17462346#comment-17462346
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #13 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/13/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/13/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/13/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/13/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461969#comment-17461969 ] Josh Elser commented on HBASE-26067: Heh! Yes, strongly agree. Let me try to write something :) > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461819#comment-17461819 ] Duo Zhang commented on HBASE-26067: --- I think we need a fat release note here :) [~elserj] [~wchevreuil]. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461726#comment-17461726
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #12 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/12/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/12/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/12/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/12/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461451#comment-17461451
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #11 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/11/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/11//console].
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/11//console].
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- Something went wrong running this stage, please [check relevant console
output|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/11//console].
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461120#comment-17461120
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #10 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/10/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17461046#comment-17461046 ] Josh Elser commented on HBASE-26067: I've just merged the book updates from Wellington. We have no more open tasks here :) Let me put together a VOTE thread (given Duo's suggestion to still do a formal vote). > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17458085#comment-17458085
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #9 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/9/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/9/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/9/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/9/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17454859#comment-17454859
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #8 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/8/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/8/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/8/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/8/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17453730#comment-17453730
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #7 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/7/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/7/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/7/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/7/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450473#comment-17450473 ] Duo Zhang commented on HBASE-26067: --- For me I think HBASE-26286 should be included. The metrics for BrokenStoreFileCleaner could be a follow on. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450453#comment-17450453
]
Wellington Chevreuil commented on HBASE-26067:
--
{quote} I think compaction speed could affect performance. {quote}
Yeah, we have seen writes been affected when compactions were running, probably
hboss locks are the main culprit here.
{quote}So I think we are close to merge this back? {quote}
Yes! Apart from the ref guide task, there are two more pending. Should we wait
for those, or can we merge those as follow on? Bulkload support is also missing
yet, but I guess we already agreed to work on it on a separate jira.
{quote}And I think we could also backport this to branch-2?{quote}
+1.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450442#comment-17450442 ] Duo Zhang commented on HBASE-26067: --- OK, sounds reasonable. I think compaction speed could affect performance. So I think we are close to merge this back? And I think we could also backport this to branch-2? Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450324#comment-17450324
]
Wellington Chevreuil commented on HBASE-26067:
--
{quote}
So we could have about 6% performance gain in YCSB? It is a bit surprise to me,
I didn't think we could increase read/write performance, we could just save
lots of time for the background tasks, such as flush and compaction...
{quote}
I think part of it is due to split storms happening during the run. Splits were
taking longer to complete on the hboss/renames cluster, causing some requests
to timeout for longer. Another observation was that the hboss/renames cluster
had more regions at the end of the run. Maybe because compactions were much
slower, the split threshold was being met more often, leading to more splits.
Snapshots recovery is another area we could see improvements, when dealing with
snapshots of tables with hundreds to thousands of regions. We are doing more
tests to consolidate numbers, will post more details here later.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450151#comment-17450151
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #6 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/General_20Nightly_20Build_20Report/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/6/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17450058#comment-17450058
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #5 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/5/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17449460#comment-17449460 ] Duo Zhang commented on HBASE-26067: --- So we could have about 6% performance gain in YCSB? It is a bit surprise to me, I didn't think we could increase read/write performance, we could just save lots of time for the background tasks, such as flush and compaction... > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448499#comment-17448499
]
Wellington Chevreuil commented on HBASE-26067:
--
{quote}Do you have any initial perf numbers on this? I'm curious how it
compares to Stephen's original results.{quote}
We have done some ycsb runs with write only workloads and compared the ycsb
throughput/latency metrics between runs with and without this feature. When
enabling the feature, we also disabled HBoss) Here's an example:
SFT disabled/HBoss enabled :
{noformat}
[OVERALL], RunTime(ms), 2556932
[OVERALL], Throughput(ops/sec), 3910.9370135772087
[TOTAL_GCS_PS_Scavenge], Count, 3660
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8642
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.3379831767133424
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 8.995155131227581E-4
[TOTAL_GCs], Count, 3661
[TOTAL_GC_TIME], Time(ms), 8665
[TOTAL_GC_TIME_%], Time(%), 0.33888269222646517
[CLEANUP], Operations, 20
[CLEANUP], AverageLatency(us), 86.9
[CLEANUP], MinLatency(us), 2
[CLEANUP], MaxLatency(us), 1337
[CLEANUP], 95thPercentileLatency(us), 236
[CLEANUP], 99thPercentileLatency(us), 1337
[INSERT], Operations, 1000
[INSERT], AverageLatency(us), 2552.2381777
[INSERT], MinLatency(us), 1146
[INSERT], MaxLatency(us), 1753087
[INSERT], 95thPercentileLatency(us), 3025
[INSERT], 99thPercentileLatency(us), 20623
[INSERT], Return=OK, 1000 {noformat}
SFT enablde/HBoss Disabled:
{noformat}
[OVERALL], RunTime(ms), 2420289
[OVERALL], Throughput(ops/sec), 4131.737986661923
[TOTAL_GCS_PS_Scavenge], Count, 3639
[TOTAL_GC_TIME_PS_Scavenge], Time(ms), 8213
[TOTAL_GC_TIME_%_PS_Scavenge], Time(%), 0.33933964084454377
[TOTAL_GCS_PS_MarkSweep], Count, 1
[TOTAL_GC_TIME_PS_MarkSweep], Time(ms), 23
[TOTAL_GC_TIME_%_PS_MarkSweep], Time(%), 9.502997369322424E-4
[TOTAL_GCs], Count, 3640
[TOTAL_GC_TIME], Time(ms), 8236
[TOTAL_GC_TIME_%], Time(%), 0.34028994058147605
[CLEANUP], Operations, 20
[CLEANUP], AverageLatency(us), 74.85
[CLEANUP], MinLatency(us), 1
[CLEANUP], MaxLatency(us), 1047
[CLEANUP], 95thPercentileLatency(us), 261
[CLEANUP], 99thPercentileLatency(us), 1047
[INSERT], Operations, 1000
[INSERT], AverageLatency(us), 2415.8930098
[INSERT], MinLatency(us), 1129
[INSERT], MaxLatency(us), 1608703
[INSERT], 95thPercentileLatency(us), 2979
[INSERT], 99thPercentileLatency(us), 19231
[INSERT], Return=OK, 1000 {noformat}
This, of course, is what's observed from the ycsb client. We had not yet
measured performance/throughput of individual internal operations that are
probably most impacted, such as compactions, splits/merges, snapshots, to name
a few.
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17448302#comment-17448302 ] Zach York commented on HBASE-26067: --- [~wchevreuil][~zhangduo] Do you have any initial perf numbers on this? I'm curious how it compares to Stephen's original results. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446816#comment-17446816
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #4 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/4/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/4/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/4/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/4/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17446530#comment-17446530
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #3 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/3/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/3/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/3/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/3/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443093#comment-17443093
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #2 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/2/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/2/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/2/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/2/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443043#comment-17443043 ] Duo Zhang commented on HBASE-26067: --- I have set up the nightly job for branch HBASE-26067. The broken first build is a known issue, do not worry. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17443042#comment-17443042
]
Hudson commented on HBASE-26067:
Results for branch HBASE-26067
[build #1 on
builds.a.o|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/1/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/1/General_20Nightly_20Build_20Report/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/1/JDK8_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(x) {color:red}-1 jdk11 hadoop3 checks{color}
-- For more information [see jdk11
report|https://ci-hadoop.apache.org/job/HBase/job/HBase%20Nightly/job/HBASE-26067/1/JDK11_20Nightly_20Build_20Report_20_28Hadoop3_29/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442834#comment-17442834 ] Wellington Chevreuil commented on HBASE-26067: -- I would like to remove our dependency on renames, but I agree changing the current archiving logic would deserve an issue on its own. Meanwhile, I had submitted HBASE-26454 to remove temp/rename from CreateTableProcedure. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442796#comment-17442796 ] Duo Zhang commented on HBASE-26067: --- It depends on our goal. If the goal here is to remove all the renames, then yes, HFile archiving is a problem. But as you said, it is a bit challenging here. With store file tracker, it is OK for us to leave the HFile there, without the actual 'archive operation'. And when we reached the TTL, we just delete the HFile directly. But the problem as you said, how can not find out the 'archived' HFiles, as in our current design, the 'archived' HFiles should be plaved under the archive directory. Changing this will need a very big refactoring. So I think we'd better open another issue for this, and collect more feedbacks to see if others in the community can have better abstraction and solution. Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17442697#comment-17442697 ] Wellington Chevreuil commented on HBASE-26067: -- Should we worry about HFile archiving? Currently, DeleteTable and GCRegion procedures rely on HFileArchiver to move region dirs/hfiles out of namespace/table dirs, to archive folder. Delete table, in case of large tables, can lead to lots of renames. I think changing current archiving logic to avoid rename would be challenging, as we would require some way to track archived tables/regions (maybe another status in meta or a new system table), and rewriting of existing hbck methods that rely on filesystem state to fix meta inconsistencies. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.20.1#820001)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[
https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415754#comment-17415754
]
Josh Elser commented on HBASE-26067:
{quote}I do not think it should be a blocker of this feature, but we do need to
redesign the bulk load processing, to make it better suit the new store file
tracker framework. Can be a follow on issue.
{quote}
Yeah, agreed. I think as long as this isn't the default implementation, that's
an acceptable thing to withhold.
I know that it took Accumulo a _very_ long time to get its storefile tracking
correct for all of the bulk loading edge cases. I suspect it will be quite
tricky :)
> Change the way on how we track store file list
> --
>
> Key: HBASE-26067
> URL: https://issues.apache.org/jira/browse/HBASE-26067
> Project: HBase
> Issue Type: Umbrella
> Components: HFile
>Reporter: Duo Zhang
>Assignee: Duo Zhang
>Priority: Major
>
> Open a separated jira to track the work since it can not be fully included in
> HBASE-24749.
> I think this could be a landed prior to HBASE-24749, as if this works, we
> could have different implementations for tracking store file list.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (HBASE-26067) Change the way on how we track store file list
[ https://issues.apache.org/jira/browse/HBASE-26067?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17415581#comment-17415581 ] Duo Zhang commented on HBASE-26067: --- I think we are close to finish this feature as we have clear solutions for most sub tasks. There is still a pending question about bulk load support. The old implementation just use rename to commit the store file, but now we need an extra step, which is inserting to store file tracker. For the default implementation, there is no problem as inserting is just a no op, but for other store file tracker implementation, such as file based store file tracker, if we fail between renaming and inserting, the users will find out that, the file has been moved away, but the store has not loaded it, the file is just gone... I do not think it should be a blocker of this feature, but we do need to redesign the bulk load processing, to make it better suit the new store file tracker framework. Can be a follow on issue. Thanks. > Change the way on how we track store file list > -- > > Key: HBASE-26067 > URL: https://issues.apache.org/jira/browse/HBASE-26067 > Project: HBase > Issue Type: Umbrella > Components: HFile >Reporter: Duo Zhang >Assignee: Duo Zhang >Priority: Major > > Open a separated jira to track the work since it can not be fully included in > HBASE-24749. > I think this could be a landed prior to HBASE-24749, as if this works, we > could have different implementations for tracking store file list. -- This message was sent by Atlassian Jira (v8.3.4#803005)
