[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-06-09 Thread stack (JIRA) 


[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16507051#comment-16507051
 ] 

stack commented on HBASE-20483:
---

Link to report comparing flushing rates before and after

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Fix For: 2.0.1
>
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-Perpetual-flush-memstore-fill-etc.-tools.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-06-08 Thread stack (JIRA) 


[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16506812#comment-16506812
 ] 

stack commented on HBASE-20483:
---

bq. The other jira abt SegmentScanner fixed this also right?

[~anoop.hbase] Sorry. Took me a while to generate the new graphs.

Yes,  HBASE-20628 SegmentScanner does over-comparing when one flushing plus 
work done on speeding up Scanning -- mainly HBASE-20620 Tighter 
ByteBufferKeyValue Cell Comparator; part 2 and part 1 -- have made it so flush 
in hbase2 is now as regular if not more so than hbase1.  We flush at about the 
same rate and the same sizes (hbase2 is tighter to the limits in my tests).

I put new results on new sheet here 
https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1517830749.
 See diagrams for runs against 4 regions, 40, and 400 comparing hbase1 to 
hbase2. See how in the sheet June07, 
https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1517830749,
 where we diagram the flushes. For how it was before, see the previous sheet, 
'Flush History', 
https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826.

Let me resolve.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-Perpetual-flush-memstore-fill-etc.-tools.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-06-06 Thread Anoop Sam John (JIRA) 


[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16504311#comment-16504311
 ] 

Anoop Sam John commented on HBASE-20483:


[~stack]  The other jira abt SegmentScanner fixed this also right?

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-Perpetual-flush-memstore-fill-etc.-tools.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-22 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16484490#comment-16484490
 ] 

stack commented on HBASE-20483:
---

Rebase of the perpetual test tools in  
[^0001-Perpetual-flush-memstore-fill-etc.-tools.patch] 

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-Perpetual-flush-memstore-fill-etc.-tools.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-14 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16475260#comment-16475260
 ] 

stack commented on HBASE-20483:
---

Rebase the perpetual tests in  [^0001-HBASE-20483-perpetual.tests.patch] 

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, 
> 0001-HBASE-20483-perpetual.tests.patch, PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-12 Thread Anastasia Braginsky (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16473266#comment-16473266
 ] 

Anastasia Braginsky commented on HBASE-20483:
-

Thank you, [~stack]! I'll try to take a look on HBASE-20411, but please do not 
wait for me. Your document about performance is very interesting, I need to get 
deeper into it.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-11 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472866#comment-16472866
 ] 

stack commented on HBASE-20483:
---

[~anastas]  HBASE-20411 removes the worst 'locking' offender in the write-path. 
There are other 'frictions' but this one overwhelmed the others. It does not 
make for much of an improvement in throughput; that is an ongoing project (IMC 
is not the problem; IMC is unable to shine, in my opinion, until we figure this 
basic perf drop-off).

Yes, MSLAB is on in 1.4.

If you want to follow-along, 
https://docs.google.com/document/d/1vZ_k6_pNR1eQxID5u1xFihuPC7FkPaJQW8c4M5eA2AQ/edit
 is where I"m doing my investigations (HBASE-20188). Thanks.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-11 Thread Anastasia Braginsky (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16472684#comment-16472684
 ] 

Anastasia Braginsky commented on HBASE-20483:
-

{quote}bq.I'm back to this issue now via HBASE-20411 and other explorations

bq.Yeah, its after. (regarding IMC disabled)
{quote}
I see that you are working without IMC and that you believe the problem is due 
to locking around the sizing (HBASE-20411). Let's see if it helps.

On the other note, is there MSLAB in the hbase 1.4 (the version you are 
comparing to)?

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-09 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16469520#comment-16469520
 ] 

stack commented on HBASE-20483:
---

Update my perpetual flush test and add a perpetual fill memstore for comparing 
branch-1.4 and branch-2.0.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: 
> 0001-HBASE-20483-perpertual-flush-and-memstore-fill-branch-2.0.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> 0001-HBASE-20483-perpetual-flush.branch-1.4.patch, 
> PerpetualFlushingProgram.patch, PerpetualFlushingProgram1.2.patch, 
> SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-08 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468342#comment-16468342
 ] 

stack commented on HBASE-20483:
---

bq. Is the above CPU usage after the PE patch where you made IMC disabled ? 
Because the above command will create table with IMC as you may be knowing 
already. Just confirming.

Yeah, its after.

hbase1 seems to flush more data and looking at inbound, there is more being 
written into hbase1 (I was thinking that replays or whatever were upping what 
was inbound on hbase2). We have few metrics in here, HBASE-20549, so hacking in 
some.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-08 Thread ramkrishna.s.vasudevan (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16468334#comment-16468334
 ] 

ramkrishna.s.vasudevan commented on HBASE-20483:


Is the above CPU usage after the PE patch where you made IMC disabled ? Because 
the above command will create table with IMC as you may be knowing already. 
Just confirming. 

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-08 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467970#comment-16467970
 ] 

stack commented on HBASE-20483:
---

Comparing 1.4 to 2.0, 1.4 flushes 70G to 2.0s 55G over the course of the test 
run.

{{./hbase/bin/hbase  --config ~/conf_hbase 
org.apache.hadoop.hbase.PerformanceEvaluation --nomapred --presplit=40 
--size=30 --columns=10 --valueSize=100 --writeToWAL=false randomWrite 100}}

Applying to memstore takes 28% of all CPU in 1.4 and almost 80% in 2.0.0. Its 
like we are double adding Cells.



> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-08 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467561#comment-16467561
 ] 

stack commented on HBASE-20483:
---

I've not tried [~anastas] I can in a while. I'm back to this issue now via 
HBASE-20411 and other explorations (helped by [~anoop.hbase] and [~ram_krish]). 
This issue is why our writes are down. We  go to the memory global limits and 
stay there, blocked, unable to clear memory as fast as we did in hbase1 because 
flushing is ~half-speed. Working on this now.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-05-08 Thread Anastasia Braginsky (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16467304#comment-16467304
 ] 

Anastasia Braginsky commented on HBASE-20483:
-

Do you get the same flush performance when working with CCM/CAM/NONE?

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-26 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454187#comment-16454187
 ] 

stack commented on HBASE-20483:
---

Attached little perpetual flushing programs so could study flush performance 
and compare to 1.2.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: PerpetualFlushingProgram.patch, 
> PerpetualFlushingProgram1.2.patch, SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-26 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16454010#comment-16454010
 ] 

stack commented on HBASE-20483:
---

Attached [~anoop.hbase]'s resort to ColelctionBackedScanner patch.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
> Attachments: SnapshotSegmentScanner_2.0.0.patch
>
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-25 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16452490#comment-16452490
 ] 

stack commented on HBASE-20483:
---

Recalibrating to repro Anoop findings by using PE loading (easier and hbase2 
client against hbase), I am still unable to repro. When I apply the 
CollectionBackedScanner patch (attached), all seizes up with 
CallQueueTooBigException. Applying above patch from yesterday, I get a slight 
lift.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-24 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451712#comment-16451712
 ] 

stack commented on HBASE-20483:
---

I tried putting in place ColelctionBackedScanner. It makes little difference 
for me on cluster runs. When I try it in my little rig that just does 
continuous flush with DisabledWALProvider, i see that flush runs a little 
faster in 2.0.0 with CollectionBackedScanner so thats good (It seems to write 
out about same number of edits -- and eventual filesize of 164M -- in about 2.1 
seconds where 1.2.7 is taking 2.4 for about same amount).

Need more wins (smile).

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-24 Thread Anoop Sam John (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451592#comment-16451592
 ] 

Anoop Sam John commented on HBASE-20483:


In 1.x we have CollectionBackedScanner scanner (very simple) for Snapshots.  
When we create a user scan on top of memstore, we have the SegmentScanner kind 
of thing. In 2.0 even on snapshot, we have the SegmentScanner.  I changed it 
back to 1.x way and seems got the same flush perf (with cluster with PE write 
workload).  And with that, the overall perf for write also almost same as that 
of 1.4.4.   There were no 512 MB memstore size breach and so sleeps at client 
and retry.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-24 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451545#comment-16451545
 ] 

stack commented on HBASE-20483:
---

I see Cell compare taking 2x the CPU. We are doing more compares or compare 
itself is slower (the methods are larger now because those if/else checking 
types). I'm looking for upped comparing count. The above patch makes the hbase2 
flush heavy CPU consumers resemble those of hbase1.

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-24 Thread stack (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451491#comment-16451491
 ] 

stack commented on HBASE-20483:
---

I assigned to you sir.

I'm working it currently. I'll attach rigs for comparing 1.2 and 2.0 flushes.

Doing the below gets rid of a load of compares done on reseek making it seem 
like flushes take the same time in 1.2. and 2.0 locally but up on cluster, I 
still lag badly in write throughput. Digging in more:

{code}
diff --git 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java
 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java
index a8b0d3d501..417083f11b 100644
--- 
a/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java
+++ 
b/hbase-server/src/main/java/org/apache/hadoop/hbase/regionserver/SegmentScanner.java
@@ -162,9 +162,20 @@ public class SegmentScanner implements KeyValueScanner {
 get it. So we remember the last keys we iterated to and restore
 the reseeked set to at least that point.
 */
+/*
 iter = getIterator(getHighest(cell, last));
 updateCurrent();
 return (current != null);
+*/
+  while(iter.hasNext()){
+Cell next = iter.next();
+int ret = segment.compare(next, cell);
+if(ret >= 0){
+  current = next;
+  return true;
+}
+  }
+  return false;
   }
{code}


> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Assignee: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (HBASE-20483) [PERFORMANCE] Flushing is 2x slower in hbase2.

2018-04-24 Thread Anoop Sam John (JIRA) 

[ 
https://issues.apache.org/jira/browse/HBASE-20483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16451448#comment-16451448
 ] 

Anoop Sam John commented on HBASE-20483:


Assign to me boss?

> [PERFORMANCE] Flushing is 2x slower in hbase2.
> --
>
> Key: HBASE-20483
> URL: https://issues.apache.org/jira/browse/HBASE-20483
> Project: HBase
>  Issue Type: Sub-task
>  Components: Performance
>Reporter: stack
>Priority: Major
>
> When flush is slow, memstores backup and go slower. See "1.2.7 vs. 2.0.0 
> Flush History" 
> https://docs.google.com/spreadsheets/d/1sihTxb4aCplR3Rr_GGXkPlwhMIm-CbB9j_5339AS0Zc/edit#gid=1016758826
>   for compare. First noted by [~anoop.hbase]



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)