[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14555277#comment-14555277 ] Stefania commented on CASSANDRA-9240: - Yes it is thank you. I did find the problem on my branch and it was something unrelated, see [this comment|https://issues.apache.org/jira/browse/CASSANDRA-8897?focusedCommentId=14543355&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14543355] on CASSANDRA-8897. Sorry for not posting here as well. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14554228#comment-14554228 ] Alan Boudreault commented on CASSANDRA-9240: [~Stefania] Is everything Ok now? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533781#comment-14533781
]
Stefania commented on CASSANDRA-9240:
-
Thanks for this. I see the mean latency on read doubles after restart on trunk,
is this expected?
On the 8897 branch it's even worse, it's already double that of trunk on the
first read and it degrades further after a restart so I definitely did
something wrong which I'll have to work out but I'll carry on the discussion on
that ticket.
Here are the results in case anyone is interested.
{code}
Mean latency (one none running in foreground with -Xms500M -Xmx500M -Xmn50M,
inserting 200k and reading back 50k):
TRUNK insert 7.2
TRUNK read 8.7
TRUNK read after restart 16.4
8897 insert 6.8
8897 read 16.6
8897 read after restart 22.3
{code}
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Benedict
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log,
> trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533746#comment-14533746
]
Alan Boudreault commented on CASSANDRA-9240:
[~Stefania] Use a low heap cassandra (ccm default in example: -Xms500M -Xmx500M
-Xmn50M) and do some write, restart cassandra and do some read like:
{code}
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(insert=1\) n=200 -rate threads=50
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(read=1\) n=5 -rate threads=50
#restart cassandra
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(read=1\) n=50 -rate threads=50
{code}
Check my [^run_issue.sh] script to see the complete test.
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Benedict
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log,
> trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14533736#comment-14533736 ] Stefania commented on CASSANDRA-9240: - [~tjake], [~benedict], [~aboudreault], this fix conflicted with CASSANDRA-8897 (no more CompressedPoolingSegmentedFile in favor or a buffer pool). Could you sum-up the easiest way to reproduce this issue so I can do some tests on the 8897 branch? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14530630#comment-14530630 ] T Jake Luciani commented on CASSANDRA-9240: --- bq. I've pushed an updated version with that changed +1, You can switch LZ4 to use direct buffers on commit if you want > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526859#comment-14526859 ] Benedict commented on CASSANDRA-9240: - Well, that's easily fixed :) I've pushed an updated version with that changed > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526681#comment-14526681 ] T Jake Luciani commented on CASSANDRA-9240: --- It is still needed because only mmap codepath uses the ByteBuffer ICompressor api the standard mode uses byte[] which won't work with direct > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526669#comment-14526669
]
Benedict commented on CASSANDRA-9240:
-
bq. But the compressors need to specify which kind of buffer they accept,
example DeflateCompressors can't use Direct...
That's the {{useDirect}} condition - the {{useMmap}} (which has been replaced
by {{chunkSegments != null}}) isn't necessary for that I don't think?
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Benedict
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log,
> trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526663#comment-14526663 ] T Jake Luciani commented on CASSANDRA-9240: --- .bq I was planning on removing the constraint to only use direct when using mmapped files in a follow up commit. The original code did this "if (useMmap && useDirect...)" But the compressors need to specify which kind of buffer they accept, example DeflateCompressors can't use Direct... > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526659#comment-14526659
]
Benedict commented on CASSANDRA-9240:
-
bq. Benedict you have a bug in the CRAR constructor. allocateBuffer() is being
called before segments are mapped so it is always creating an on heap buffer.
Good catch. I was planning on removing the constraint to only use direct when
using mmapped files in a follow up commit. Any opposition to doing that here
(since it also fixes that)?
bq. Minor nit in your Cleanup method. For multiple levels of conditionals and
loops please add {}
I think we should clarify our code style policy on this, since I do this kind
of thing a lot, certainly at the very least for multi-nested loops or
conditionals, and for conditionals inside of loops, but a loop nested inside a
conditional is its own unique case. Would be good to have an official policy.
Could you and the other PMCs (or whoever decides this stuff) come up with an
official position on when we can and cannot omit braces?
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Benedict
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log,
> trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14526647#comment-14526647
]
T Jake Luciani commented on CASSANDRA-9240:
---
[~benedict] you have a bug in the CRAR constructor. allocateBuffer() is being
called before segments are mapped so it is always creating an on heap buffer.
Minor nit in your Cleanup method. For multiple levels of conditionals and loops
please add {}
Finally, I know CASSANDRA-8897 is going to remove pooling, but how will we
avoid the cost of un-mapping the files if we don't pool?
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: Benedict
>Priority: Minor
> Fix For: 3.x
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log,
> trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14525911#comment-14525911 ] Alan Boudreault commented on CASSANDRA-9240: Thanks [~benedict]. I tested your branch and it works great. (also tried disk_access_mode: standard). > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14524374#comment-14524374 ] Benedict commented on CASSANDRA-9240: - OK, so I've pushed a fix for this [here|https://github.com/belliottsmith/cassandra/tree/9240] This both reintroduces the pooling, and shares the mapped segments between all readers, for both pooled and regular compressed files. This could be tidied up, but since we're eliminating a lot of the functionality in these classes in the near future time is probably better spent cleaning them up as that happens. FTR, I managed to reproduce the problem with a regular heap with mmap enabled on trunk, and could eliminate the problem without reintroducing pooling, at least with those heap settings. So the cleaning of the buffers was likely the main culprit. I could strangely reproduce some reduced throughput on trunk with standard mode without pooling vs with, but not on my branch. That's a little odd, but probably a rabbit hole not worth diving into. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523942#comment-14523942 ] Benedict commented on CASSANDRA-9240: - But the pooling is (or should be) effectively a no-op with mmap off. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523880#comment-14523880 ] T Jake Luciani commented on CASSANDRA-9240: --- It's something about the pooling itself... > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523875#comment-14523875 ] Benedict commented on CASSANDRA-9240: - Well now that really doesn't make sense, since it should do nothing without mmap. It won't open/close the channel, since that's wrapped by the SegmentedFile, and it uses a heap buffer, so it cannot waste time cleaning it > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523829#comment-14523829 ] T Jake Luciani commented on CASSANDRA-9240: --- I also set disk_access_mode: standard (so not mmap) and it was also 50% slower > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523777#comment-14523777 ] Benedict commented on CASSANDRA-9240: - CompressedPoolingSegmentedFile != CompressedSegmentedFile Doh. But it's interesting, and has demonstrated some unexpected behaviour, since this seems to suggest there could be serious (atypical) performance degradation from an insufficiently sized cache. It _seems_ from the profile that we're incurring costs on reading from the buffer more than anywhere else. Which seems to suggest it may result in the OS actually dropping the pages from its cache. At the very least some cost incurred after remapping is high, in which case calling the cleaner isn't probably a good idea. It seems like the safest thing is to introduce a specialized SegmentedFile for mmapped compressed files, that retains all of the mapped sections to pass into the CompressedRandomAccessReader on construction, so that these costs are guaranteed only ever to occur once per sstable, and not once per reader. This is possible thanks to the recent work of [~stefania_alborghetti], and I think we should exploit it here to ensure this isn't a problem even after my boo-boo is fixed. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523690#comment-14523690 ] T Jake Luciani commented on CASSANDRA-9240: --- I bisected the issue to CASSANDRA-9112 Not sure what could cause the issue there? [~benedict] > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523367#comment-14523367 ] T Jake Luciani commented on CASSANDRA-9240: --- I've gone back to CASSANDRA-8464 and I don't see this issue. Going to start bisecting. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523310#comment-14523310 ] Alan Boudreault commented on CASSANDRA-9240: oh right. Sorry, I had forgotten I tested that a few days ago. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523288#comment-14523288 ] T Jake Luciani commented on CASSANDRA-9240: --- Use a larger heap? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14523287#comment-14523287 ] Alan Boudreault commented on CASSANDRA-9240: Is there a way to by-pass this direct buffer slowdown? So I could restart my compaction tests. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.x > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517462#comment-14517462 ] T Jake Luciani commented on CASSANDRA-9240: --- It's worse with small heap. No issue with large heap. Also with no mmap access it's still getting progressively slower with a small heap which is strange. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517321#comment-14517321 ] Benedict commented on CASSANDRA-9240: - Maybe because a larger heap lets it avoid flushing (or flushing as often)? Or do you mean after a restart with a larger heap it is still fine? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517315#comment-14517315 ] T Jake Luciani commented on CASSANDRA-9240: --- So looks like its happening only with the following low heap settings: -Xms500M -Xmx500M -Xmn50M > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, runs.log, > trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517098#comment-14517098 ] Alan Boudreault commented on CASSANDRA-9240: run after run. Did two subsequent runs of n=50, both was bad perf. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517065#comment-14517065 ] T Jake Luciani commented on CASSANDRA-9240: --- [~aboudreault] does the issue happen consistently run after run? or just right after a restart? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517053#comment-14517053 ] Alan Boudreault commented on CASSANDRA-9240: no, on a Ubuntu trusty. I reproduced that issue on my laptop and a ctool instance (ubuntu precise). > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani >Priority: Minor > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[
https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14517052#comment-14517052
]
T Jake Luciani commented on CASSANDRA-9240:
---
I just tried reproducing with the following:
{code}
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(insert=1\) n=200 -rate threads=50
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(read=1\) n=5 -rate threads=50
#restart cassandra
./tools/bin/cassandra-stress user profile=/home/jake/Downloads/issue.yaml
ops\(read=1\) n=5 -rate threads=50
{code}
Both read runs yielded 6-7k ops/sec.
[~aboudreault] Are you on a mac maybe?
> Performance issue after a restart
> -
>
> Key: CASSANDRA-9240
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9240
> Project: Cassandra
> Issue Type: Bug
>Reporter: Alan Boudreault
>Assignee: T Jake Luciani
>Priority: Minor
> Fix For: 3.0
>
> Attachments: Cassandra.snapshots.zip,
> cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log,
> cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log,
> cassandra_trunk-clientrequest-read.log, cassandra_trunk.log,
> cassandra_trunk_no_restart-clientrequest-read.log,
> cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql
>
>
> I have noticed a performance issue while I was working on compaction perf
> tests for CASSANDRA-7409. The performance for my use case is very bad after a
> restart. It is mostly a read performance issue but not strictly. I have
> attached my use case (see run_issue.sh and issue.yaml) and all test logs for
> 2.1.4, 2.1.5 and trunk:
> * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k
> ops/second and ~2-2.5k of read latency.
> * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency.
> * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5.
> EDIT: branch cassandra-2.1 is OK.
> I can help to bisect and/or profile on Monday if needed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516958#comment-14516958 ] Benedict commented on CASSANDRA-9240: - bq. Could it be that the OS is evicting memory-mapped pages No, definitely not. But you're on the right track. I'm pretty sure the problem is that we explicitly "clean" the buffers on "deallocating" the RARs. If the reader is memory-mapped this likely results in the OS evicting at least those pages (perhaps more?) Reassigning to [~tjake] since this is a problem from CASSANDRA-8464 itself, and not the new lz4 methods. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Branimir Lambov > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516769#comment-14516769 ] Branimir Lambov commented on CASSANDRA-9240: Before 8464 reads from disk were explicit, after 8464 the data is memory-mapped hence their time share is accounted for in whatever happens to be reading the data. In this case that's LZ4Compressor.uncompress. More time spent in it now means data being read more slowly. Could it be that the OS is evicting memory-mapped pages from cache when the application that owns them closes? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Branimir Lambov > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14516001#comment-14516001 ] Benedict commented on CASSANDRA-9240: - This appears to be a performance regression caused by CASSANDRA-8464. [~blambov]: from the yourkit profiles attached by [~aboudreault] it looks like it may be the native ByteBuffer method is somehow significantly slower. Since this is based on CASSANDRA-7039, could you take a look? > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: T Jake Luciani > Fix For: 3.0 > > Attachments: Cassandra.snapshots.zip, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_trunk-clientrequest-read.log, cassandra_trunk.log, > cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5 and trunk: > * 2.1.* are OK (although 2.1.4 seems to be better than 2.1.5?): ~6-7k > ops/second and ~2-2.5k of read latency. > * trunk is NOT OK: ~1.5-2k ops/second and 25-30k of read latency. > * trunk is OK without a restart: ~ same perf than 2.1.4 and 2.1.5. > EDIT: branch cassandra-2.1 is OK. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514174#comment-14514174 ] Alan Boudreault commented on CASSANDRA-9240: Yes, I can do that. I am going to profile the executions. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict > Fix For: 3.0 > > Attachments: cassandra_2.1-clientrequest-read.log, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_2.1.log, cassandra_trunk-clientrequest-read.log, > cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5, cassandra-2.1 and trunk. > 2.1.4 and 2.1.5 are OK (although 2.1.4 seems to be better?): ~6-7k ops/second > and ~2-2.5k of read latency. > branch 2.1 and trunk are NOT OK: ~1.5-2k ops/second and 25-30k of read > latency. > branch 2.1 and trunk are OK without a restart: ~ same perf than 2.1.4 and > 2.1.5. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514169#comment-14514169 ] Benedict commented on CASSANDRA-9240: - If you could grab a profile (sampled time in each method) of each situation, it could help. I would guess the problem is triggered by the creation of an sstable after commit log replay, but exactly where the increased costs are being incurred (on merge, on sstable read, whatever) is tough to say. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault >Assignee: Benedict > Fix For: 3.0 > > Attachments: cassandra_2.1-clientrequest-read.log, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_2.1.log, cassandra_trunk-clientrequest-read.log, > cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5, cassandra-2.1 and trunk. > 2.1.4 and 2.1.5 are OK (although 2.1.4 seems to be better?): ~6-7k ops/second > and ~2-2.5k of read latency. > branch 2.1 and trunk are NOT OK: ~1.5-2k ops/second and 25-30k of read > latency. > branch 2.1 and trunk are OK without a restart: ~ same perf than 2.1.4 and > 2.1.5. > I can help to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9240) Performance issue after a restart
[ https://issues.apache.org/jira/browse/CASSANDRA-9240?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514115#comment-14514115 ] Alan Boudreault commented on CASSANDRA-9240: [~benedict] Let me know if you can take a look at this and if you need anything else. > Performance issue after a restart > - > > Key: CASSANDRA-9240 > URL: https://issues.apache.org/jira/browse/CASSANDRA-9240 > Project: Cassandra > Issue Type: Bug >Reporter: Alan Boudreault > Fix For: 3.0 > > Attachments: cassandra_2.1-clientrequest-read.log, > cassandra_2.1.4-clientrequest-read.log, cassandra_2.1.4.log, > cassandra_2.1.5-clientrequest-read.log, cassandra_2.1.5.log, > cassandra_2.1.log, cassandra_trunk-clientrequest-read.log, > cassandra_trunk.log, cassandra_trunk_no_restart-clientrequest-read.log, > cassandra_trunk_no_restart.log, issue.yaml, run_issue.sh, trace_query.cql > > > I have noticed a performance issue while I was working on compaction perf > tests for CASSANDRA-7409. The performance for my use case is very bad after a > restart. It is mostly a read performance issue but not strictly. I have > attached my use case (see run_issue.sh and issue.yaml) and all test logs for > 2.1.4, 2.1.5, cassandra-2.1 and trunk. > 2.1.4 and 2.1.5 are OK (although 2.1.4 seems to be better?): ~6-7k ops/second > and ~2-2.5k of read latency. > branch 2.1 and trunk are NOT OK: ~1.5-2k ops/second and 25-30k of read > latency. > branch 2.1 and trunk are OK without a restart: ~ same perf than 2.1.4 and > 2.1.5. > I can to bisect and/or profile on Monday if needed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
