[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752895#comment-16752895
]
Hudson commented on HBASE-21738:
Results for branch branch-1.4
[build #644 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/644/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/644//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/644//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.4/644//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752775#comment-16752775
]
Hudson commented on HBASE-21738:
Results for branch branch-1.3
[build #626 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/626/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/626//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/626//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1.3/626//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[ https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16752576#comment-16752576 ] Hudson commented on HBASE-21738: SUCCESS: Integrated in Jenkins build HBase-1.3-IT #522 (See [https://builds.apache.org/job/HBase-1.3-IT/522/]) HBASE-21748 Port HBASE-21738 (Remove all the CLSM#size operation in our (apurtell: [https://github.com/apache/hbase/commit/ea3dcbcea4cc4470b683bee64df482400cec82df]) * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestCellSkipListSet.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/io/TestHeapSize.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestMemStoreChunkPool.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/CellSkipListSet.java * (edit) hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/DefaultMemStore.java * (edit) hbase-server/src/test/java/org/apache/hadoop/hbase/regionserver/TestDefaultMemStore.java > Remove all the CSLM#size operation in our memstore because it's an quite time > consuming. > > > Key: HBASE-21738 > URL: https://issues.apache.org/jira/browse/HBASE-21738 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch, > HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch, > image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png > > > Made some performance test for 100% put case in branch-2 before. > We can see that there are many latency peak in p999 latency curve , and the > peak time are almost the point time which our region is flushing. > See the [hbase20-ssd-put-100-rows-latencys-and-qps > |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png] > And, I used the > [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch] > to log some time consuming when we grab the update.writeLock() to make a > memstore snapshot. Tested again, I found those logs in [log.txt. > |https://issues.apache.org/jira/secure/attachment/12955343/log.txt] > Seems most of the time was consumed when taking memstore snapshot.. Let me > dig into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16750942#comment-16750942
]
Hudson commented on HBASE-21738:
Results for branch branch-1
[build #650 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/650/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(x) {color:red}-1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/650//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk7 checks{color}
-- For more information [see jdk7
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/650//JDK7_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-1/650//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 source release artifact{color}
-- See build output for details.
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748034#comment-16748034
]
Hudson commented on HBASE-21738:
Results for branch branch-2
[build #1625 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625/]:
(x) *{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2/1625//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16748005#comment-16748005
]
Hudson commented on HBASE-21738:
Results for branch branch-2.0
[build #1275 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.0/1275//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747989#comment-16747989
]
Hudson commented on HBASE-21738:
Results for branch master
[build #736 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/master/736/]: (x)
*{color:red}-1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/master/736//General_Nightly_Build_Report/]
(x) {color:red}-1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/736//JDK8_Nightly_Build_Report_(Hadoop2)/]
(x) {color:red}-1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/master/736//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[
https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747982#comment-16747982
]
Hudson commented on HBASE-21738:
Results for branch branch-2.1
[build #791 on
builds.a.o|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/791/]:
(/) *{color:green}+1 overall{color}*
details (if available):
(/) {color:green}+1 general checks{color}
-- For more information [see general
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/791//General_Nightly_Build_Report/]
(/) {color:green}+1 jdk8 hadoop2 checks{color}
-- For more information [see jdk8 (hadoop2)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/791//JDK8_Nightly_Build_Report_(Hadoop2)/]
(/) {color:green}+1 jdk8 hadoop3 checks{color}
-- For more information [see jdk8 (hadoop3)
report|https://builds.apache.org/job/HBase%20Nightly/job/branch-2.1/791//JDK8_Nightly_Build_Report_(Hadoop3)/]
(/) {color:green}+1 source release artifact{color}
-- See build output for details.
(/) {color:green}+1 client integration test{color}
> Remove all the CSLM#size operation in our memstore because it's an quite time
> consuming.
>
>
> Key: HBASE-21738
> URL: https://issues.apache.org/jira/browse/HBASE-21738
> Project: HBase
> Issue Type: Sub-task
> Components: Performance
>Reporter: Zheng Hu
>Assignee: Zheng Hu
>Priority: Critical
> Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5
>
> Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch,
> HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch,
> image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png
>
>
> Made some performance test for 100% put case in branch-2 before.
> We can see that there are many latency peak in p999 latency curve , and the
> peak time are almost the point time which our region is flushing.
> See the [hbase20-ssd-put-100-rows-latencys-and-qps
> |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png]
> And, I used the
> [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch]
> to log some time consuming when we grab the update.writeLock() to make a
> memstore snapshot. Tested again, I found those logs in [log.txt.
> |https://issues.apache.org/jira/secure/attachment/12955343/log.txt]
> Seems most of the time was consumed when taking memstore snapshot.. Let me
> dig into this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[ https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747958#comment-16747958 ] Zheng Hu commented on HBASE-21738: -- bq. Even in Segment also.. 1.x we were not having such locked areas in the hot write path.. Anyway will come back to that. The other part such as updatesLock & AtomicReference, is only related to in-memory compaction, no impact on the default memstore. > Remove all the CSLM#size operation in our memstore because it's an quite time > consuming. > > > Key: HBASE-21738 > URL: https://issues.apache.org/jira/browse/HBASE-21738 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch, > HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch, > image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png > > > Made some performance test for 100% put case in branch-2 before. > We can see that there are many latency peak in p999 latency curve , and the > peak time are almost the point time which our region is flushing. > See the [hbase20-ssd-put-100-rows-latencys-and-qps > |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png] > And, I used the > [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch] > to log some time consuming when we grab the update.writeLock() to make a > memstore snapshot. Tested again, I found those logs in [log.txt. > |https://issues.apache.org/jira/secure/attachment/12955343/log.txt] > Seems most of the time was consumed when taking memstore snapshot.. Let me > dig into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[ https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747902#comment-16747902 ] Anoop Sam John commented on HBASE-21738: bq.I've seen that, it's ThreadSafeMemStoreSizing. Maybe use a LongAdder instead of AtomicXXX ? Not just that.. Even in Segment also.. 1.x we were not having such locked areas in the hot write path.. Anyway will come back to that. > Remove all the CSLM#size operation in our memstore because it's an quite time > consuming. > > > Key: HBASE-21738 > URL: https://issues.apache.org/jira/browse/HBASE-21738 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch, > HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch, > image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png > > > Made some performance test for 100% put case in branch-2 before. > We can see that there are many latency peak in p999 latency curve , and the > peak time are almost the point time which our region is flushing. > See the [hbase20-ssd-put-100-rows-latencys-and-qps > |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png] > And, I used the > [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch] > to log some time consuming when we grab the update.writeLock() to make a > memstore snapshot. Tested again, I found those logs in [log.txt. > |https://issues.apache.org/jira/secure/attachment/12955343/log.txt] > Seems most of the time was consumed when taking memstore snapshot.. Let me > dig into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[ https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747809#comment-16747809 ] Zheng Hu commented on HBASE-21738: -- Pushed to branch-2/branch-2.0/branch-2.1/master, Thanks all for reviewing. > Remove all the CSLM#size operation in our memstore because it's an quite time > consuming. > > > Key: HBASE-21738 > URL: https://issues.apache.org/jira/browse/HBASE-21738 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch, > HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch, > image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png > > > Made some performance test for 100% put case in branch-2 before. > We can see that there are many latency peak in p999 latency curve , and the > peak time are almost the point time which our region is flushing. > See the [hbase20-ssd-put-100-rows-latencys-and-qps > |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png] > And, I used the > [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch] > to log some time consuming when we grab the update.writeLock() to make a > memstore snapshot. Tested again, I found those logs in [log.txt. > |https://issues.apache.org/jira/secure/attachment/12955343/log.txt] > Seems most of the time was consumed when taking memstore snapshot.. Let me > dig into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (HBASE-21738) Remove all the CSLM#size operation in our memstore because it's an quite time consuming.
[ https://issues.apache.org/jira/browse/HBASE-21738?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16747712#comment-16747712 ] Zheng Hu commented on HBASE-21738: -- bq. BTW compared to 1.x based write flow, we have now some synchronized blocks for the size accounting in Segment level or so. Would be better to eliminate that also I've seen that, it's ThreadSafeMemStoreSizing. Maybe use a LongAdder instead of AtomicXXX ? > Remove all the CSLM#size operation in our memstore because it's an quite time > consuming. > > > Key: HBASE-21738 > URL: https://issues.apache.org/jira/browse/HBASE-21738 > Project: HBase > Issue Type: Sub-task > Components: Performance >Reporter: Zheng Hu >Assignee: Zheng Hu >Priority: Critical > Fix For: 3.0.0, 2.2.0, 2.1.3, 2.0.5 > > Attachments: HBASE-21738.v1.patch, HBASE-21738.v2.patch, > HBASE-21738.v3.patch, HBASE-21738.v4.patch, add-some-log.patch, > image-2019-01-18-14-03-28-662.png, log.txt, performance-after-the-patch.v3.png > > > Made some performance test for 100% put case in branch-2 before. > We can see that there are many latency peak in p999 latency curve , and the > peak time are almost the point time which our region is flushing. > See the [hbase20-ssd-put-100-rows-latencys-and-qps > |https://issues.apache.org/jira/secure/attachment/12955341/12955341_image-2019-01-18-14-03-28-662.png] > And, I used the > [add-some-log.patch|https://issues.apache.org/jira/secure/attachment/12955342/add-some-log.patch] > to log some time consuming when we grab the update.writeLock() to make a > memstore snapshot. Tested again, I found those logs in [log.txt. > |https://issues.apache.org/jira/secure/attachment/12955343/log.txt] > Seems most of the time was consumed when taking memstore snapshot.. Let me > dig into this. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
