[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[
https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16546503#comment-16546503
]
Yu Li edited comment on HBASE-20411 at 7/17/18 12:11 PM:
-
Noticed when checking 2.1.0RC1 compatibility report, that in this JIRA we have
changed the visibility of {{MemStoreSize#constructors}} from public to
package-private. Since {{MemStoreSize}} is not marked as IS.Evolving, and in
our [refguide matrix|http://hbase.apache.org/book.html#hbase.versioning] we
have below content, are we breaking the compatibility here? Or could you help
correct my understanding here boss [~stack]? Thanks.
!compatibility-matrix.png!
was (Author: carp84):
Noticed when checking 2.1.0RC1 compatibility report, that in this JIRA we have
changed the visibility of {{MemStoreSize#constructors}} from public to
package-private. Since {{MemStoreSize}} is not marked as IS.Evolving, and in
our [refguide matrix|http://hbase.apache.org/book.html#hbase.versioning] we
have below content:
!compatibility-matrix.png!
> Ameliorate MutableSegment synchronize
> -
>
> Key: HBASE-20411
> URL: https://issues.apache.org/jira/browse/HBASE-20411
> Project: HBase
> Issue Type: Bug
>Reporter: stack
>Assignee: huaxiang sun
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: 2.load.patched.17704.lock.svg,
> 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg,
> 2.pe.write.32026.lock.svg, 2.pe.write.ameliorate.106553.lock.svg,
> 41901.lock.svg, HBASE-20411-atomiclong-longadder.patch,
> HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch,
> HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch,
> HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch,
> HBASE-20411.branch-2.0.007.patch, HBASE-20411.branch-2.0.008.patch,
> HBASE-20411.branch-2.0.009.patch, HBASE-20411.branch-2.0.010.patch,
> HBASE-20411.branch-2.0.011.patch, HBASE-20411.branch-2.0.012.patch,
> HBASE-20411.branch-2.0.013.patch, compatibility-matrix.png,
> jmc6.write_time_locks.png
>
>
> This item is migrated from HBASE-20236 so it gets dedicated issue.
> Let me upload evidence that has this synchronize as a stake in our write-time
> perf. I'll migrate the patch I posted with updates that come of comments
> posted by [~mdrob] on the HBASE-20236 issue.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16471505#comment-16471505 ] stack edited comment on HBASE-20411 at 5/11/18 5:24 AM: .014 Address [~mdrob] feedback and rebase. was (Author: stack): .014 Address [~mdrob] feedback. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, > 2.pe.write.32026.lock.svg, 2.pe.write.ameliorate.106553.lock.svg, > 41901.lock.svg, HBASE-20411-atomiclong-longadder.patch, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch, > HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch, > HBASE-20411.branch-2.0.007.patch, HBASE-20411.branch-2.0.008.patch, > HBASE-20411.branch-2.0.009.patch, HBASE-20411.branch-2.0.010.patch, > HBASE-20411.branch-2.0.011.patch, HBASE-20411.branch-2.0.012.patch, > HBASE-20411.branch-2.0.013.patch, jmc6.write_time_locks.png > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16449300#comment-16449300 ] huaxiang sun edited comment on HBASE-20411 at 4/24/18 5:17 AM: --- I have a patch ready which uses LongAdder for counters in Segment. For the counters in Segment, write/update happens in the data path, but the read happens in the control path. So LongAdder seems a perfect solution. For counters in HRegion, both write/read happens in the data path, so my patch falls back to the original AtomicLong approach. I tried to reproduce the issue with [~anoop.hbase]'s PE command, but the lock flamegraph does not show the same pattern. Will try to reproduce and report back. May come up with jmh test and the result. was (Author: huaxiang): I have a patch ready which uses LongAdder for counters in Segment. For the counters in Segment, write/update happens in the data path, but the read happens in the control path. So LongAdder seems a perfect solution. For counters in HRegion, both write/read happens in the data path, so my patch falls back to the original AtomicLong approach. I tried to reproduce the issue with [~anoop.hbase]'s PE command, but the lock flamegraph cannot reproduce the same pattern. Will try to reproduce and report back. May come up with jmh test and the result. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch, > HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch, > HBASE-20411.branch-2.0.007.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439840#comment-16439840 ] stack edited comment on HBASE-20411 at 4/16/18 6:34 PM: 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed (It does not show in this svg and in jfr, it no longer shows up in events or in 'contention' list). We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again <= This statement is wrong; in 1.2.7 our locking profile does not have the rpc semaphore; it has the mvcc completion and then a bunch of blocking in dfsclient that we don't have in 2.0.0). Not much by way of throughput improvement after this goes in. was (Author: stack): 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed. We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again <= This statement is wrong; in 1.2.7 our locking profile does not have the rpc semaphore; it has the mvcc completion and then a bunch of blocking in dfsclient that we don't have in 2.0.0). Not much by way of throughput improvement after this goes in. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch, > HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch, > HBASE-20411.branch-2.0.007.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439840#comment-16439840 ] stack edited comment on HBASE-20411 at 4/16/18 6:32 PM: 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed. We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again <= This statement is wrong; in 1.2.7 our locking profile does not have the rpc semaphore; it has the mvcc completion and then a bunch of blocking in dfsclient that we don't have in 2.0.0). Not much by way of perf improvement after this goes in. was (Author: stack): 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed. We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again). Not much by way of perf improvement though. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch, > HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch, > HBASE-20411.branch-2.0.007.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (HBASE-20411) Ameliorate MutableSegment synchronize
[ https://issues.apache.org/jira/browse/HBASE-20411?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16439840#comment-16439840 ] stack edited comment on HBASE-20411 at 4/16/18 6:32 PM: 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed. We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again <= This statement is wrong; in 1.2.7 our locking profile does not have the rpc semaphore; it has the mvcc completion and then a bunch of blocking in dfsclient that we don't have in 2.0.0). Not much by way of throughput improvement after this goes in. was (Author: stack): 2.simple.patch.69074.lock.svg is what our locking profile looks like after this patch has been applied. Blocking on MutableSegment has been removed. We are left with the Semaphore on RPC scheduling and mvcc completion (This makes our locking profile looks like 1.2.7 again <= This statement is wrong; in 1.2.7 our locking profile does not have the rpc semaphore; it has the mvcc completion and then a bunch of blocking in dfsclient that we don't have in 2.0.0). Not much by way of perf improvement after this goes in. > Ameliorate MutableSegment synchronize > - > > Key: HBASE-20411 > URL: https://issues.apache.org/jira/browse/HBASE-20411 > Project: HBase > Issue Type: Bug >Reporter: stack >Priority: Major > Attachments: 2.load.patched.17704.lock.svg, > 2.load.patched.2.17704.lock.svg, 2.more.patch.12010.lock.svg, 41901.lock.svg, > HBASE-20411.branch-2.0.001.patch, HBASE-20411.branch-2.0.002.patch, > HBASE-20411.branch-2.0.003.patch, HBASE-20411.branch-2.0.004.patch, > HBASE-20411.branch-2.0.005.patch, HBASE-20411.branch-2.0.006.patch, > HBASE-20411.branch-2.0.007.patch > > > This item is migrated from HBASE-20236 so it gets dedicated issue. > Let me upload evidence that has this synchronize as a stake in our write-time > perf. I'll migrate the patch I posted with updates that come of comments > posted by [~mdrob] on the HBASE-20236 issue. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
