[jira] [Comment Edited] (CASSANDRA-11697) Improve Compaction Throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382592#comment-15382592
]
T Jake Luciani edited comment on CASSANDRA-11697 at 7/18/16 5:01 PM:
-
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{code}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = {'sstable_compression':'LZ4Compressor'} AND compaction =
{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'}'
{code}
Each user contains 1000 pic ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png!
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
was (Author: tjake):
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{code}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = {'sstable_compression':'LZ4Compressor'} AND compaction =
{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'}'
{code}
Each picid contains 1000 value ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png!
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
> Improve Compaction Throughput
> -
>
> Key: CASSANDRA-11697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11697
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
> Attachments: image.png
>
>
> The goal of this ticket is to improve/understand the bottlenecks during
> compactions. At a high level this will involve:
> * A test system for measuring compaction time for different workloads and
> compaction strategies.
> * Profiling and analysis
> * Make improvements
> * Add throughput regression tests so we can track
> We have a lot of random tickets that relate to this so I'll link them to this
> ticket
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11697) Improve Compaction Throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382592#comment-15382592
]
T Jake Luciani edited comment on CASSANDRA-11697 at 7/18/16 4:59 PM:
-
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{code}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = {'sstable_compression':'LZ4Compressor'} AND compaction =
{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'}'
{code}
Each picid contains 1000 value ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png!
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
was (Author: tjake):
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{code}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = \{'sstable_compression':'LZ4Compressor'\} AND compaction
= \{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'\};
{code}
Each picid contains 1000 value ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
> Improve Compaction Throughput
> -
>
> Key: CASSANDRA-11697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11697
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
> Attachments: image.png
>
>
> The goal of this ticket is to improve/understand the bottlenecks during
> compactions. At a high level this will involve:
> * A test system for measuring compaction time for different workloads and
> compaction strategies.
> * Profiling and analysis
> * Make improvements
> * Add throughput regression tests so we can track
> We have a lot of random tickets that relate to this so I'll link them to this
> ticket
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11697) Improve Compaction Throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15382592#comment-15382592
]
T Jake Luciani edited comment on CASSANDRA-11697 at 7/18/16 4:58 PM:
-
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{code}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = \{'sstable_compression':'LZ4Compressor'\} AND compaction
= \{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'\};
{code}
Each picid contains 1000 value ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
was (Author: tjake):
*The state of compaction and the potential improvements*
To continue the above comment here is information on a specific schema I'm
focused on.
{quote}
CREATE TABLE userpics (
userid bigint,
picid bigint,
valueid bigint,
PRIMARY KEY(userid, picid)
) WITH compression = {'sstable_compression':'LZ4Compressor'} AND compaction =
{'class' : 'LeveledCompactionStrategy', 'sstable_size_in_mb' : '64'};
{quote}
Each picid contains 1000 value ids.
I've run this through two scenarios: Standalone vs Stressed. *Standalone* is
with compaction stress that shows the uncontested throughput. *Stressed* is
compaction running in C* and taking many writes so the CPUs are loaded.
These scenarios were run against 2.1, 3.0, trunk, and the current patched
version of compaction-perf work I've been doing.
!image.png
The *Y* axis is the number of cells/sec.
You can see that we there is a significant drop when the server is loaded.
When compactions are backing up this is the worst thing to happen.
>From 2.1 to 3.0 we improved stressed compaction 2x mainly from
>(CASSANDRA-8988, CASSANDRA-8920, CASSANDRA-8915)
But there is still a huge throughput gap we need to fix. Currently, I've made
changes to disable thread priority on compaction threads and profiling fixes to
avoid cpu time that improves throughput another 2x but the final goal is to
close this gap as much as possible as well as fix any major cpu issues on the
read and write path since those affect compaction directly as well e.g.
(CASSANDRA-10309).
As mentioned earlier I'm working on a POC to deal with compaction threads under
load that I hope will address this issue. My goal is to have something that
closes the gap as much as possibly by 3.10
> Improve Compaction Throughput
> -
>
> Key: CASSANDRA-11697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11697
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
> Attachments: image.png
>
>
> The goal of this ticket is to improve/understand the bottlenecks during
> compactions. At a high level this will involve:
> * A test system for measuring compaction time for different workloads and
> compaction strategies.
> * Profiling and analysis
> * Make improvements
> * Add throughput regression tests so we can track
> We have a lot of random tickets that relate to this so I'll link them to this
> ticket
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (CASSANDRA-11697) Improve Compaction Throughput
[
https://issues.apache.org/jira/browse/CASSANDRA-11697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15365050#comment-15365050
]
T Jake Luciani edited comment on CASSANDRA-11697 at 7/6/16 8:53 PM:
Just a general note about my findings so far:
There seems to be some conflation about the performance of compaction mainly
that low IO is slower than high IO. Our compaction is primarily CPU bound and
since we compress by default it can appear that compaction is going slow but
it's actually doing ~the same rows/sec as uncompressed. Here is an example of
compressed vs uncompressed notice the new log message from CASSANDRA-10805
shows ~ the same rows/sec.
{quote}
#UNCOMPRESSED
INFO 20:22:01 Compacted (3206a030-43b7-11e6-bf87-63e169949442) 2 sstables to
[/home/jake/workspace/cassandra/build/test/cassandra/data/cql_keyspace/table1-041d8b2043b711e6bf8763e169949442/mb-3-big,]
to level=0.
782.013MiB to 361.930MiB (~46% of original) in 43,211ms.
Read Throughput = 18.097MiB/s, Write Throughput = 8.376MiB/s,
Row Throughput = ~454,545/s. 20,000,000 total partitions merged to
10,000,000. Partition merge counts were \{2:1000, \}
#COMPRESSED
INFO 20:27:49 Compacted (00d23b40-43b8-11e6-ad90-8f9393166cb4) 2 sstables to
[/home/jake/workspace/cassandra/build/test/cassandra/data/cql_keyspace/table1-da46ad8043b711e6ad908f9393166cb4/mb-3-big,]
to level=0.
123.201MiB to 57.405MiB (~46% of original) in 44,527ms. Read Throughput =
2.767MiB/s, Write Throughput = 1.289MiB/s,
Row Throughput = ~444,444/s. 20,000,000 total partitions merged to
10,000,000. Partition merge counts were \{2:1000,\}
{quote}
*I'd like to start talking about compaction in terms of row throughput vs
MB/sec.*
There are some things we can do to make our current compaction faster like
CASSANDRA-10309 should give us a ~10% boost but the *bigger* issue and what I
think most users are seeing in the real world is the fact that under load
compaction effectively grinds to a halt. I can easily produce a 10x drop in
throughput by applying load during a regular compaction. This to me is the
problem we should be focused on since we can see this when looking at sample
user logs. I have a POC I'm working on to address this that I'll be presenting
soon.
was (Author: tjake):
Just a general note about my findings so far:
There seems to be some conflation about the performance of compaction mainly
that low IO is slower than high IO. Our compaction is primarily CPU bound and
since we compress by default it can appear that compaction is going slow but
it's actually doing ~the same rows/sec as uncompressed. Here is an example of
compressed vs uncompressed notice the new log message from CASSANDRA-10805
shows ~ the same rows/sec.
{quote}
#UNCOMPRESSED
INFO 20:22:01 Compacted (3206a030-43b7-11e6-bf87-63e169949442) 2 sstables to
[/home/jake/workspace/cassandra/build/test/cassandra/data/cql_keyspace/table1-041d8b2043b711e6bf8763e169949442/mb-3-big,]
to level=0.
782.013MiB to 361.930MiB (~46% of original) in 43,211ms.
Read Throughput = 18.097MiB/s, Write Throughput = 8.376MiB/s,
Row Throughput = ~454,545/s. 20,000,000 total partitions merged to
10,000,000. Partition merge counts were {2:1000, }
#COMPRESSED
INFO 20:27:49 Compacted (00d23b40-43b8-11e6-ad90-8f9393166cb4) 2 sstables to
[/home/jake/workspace/cassandra/build/test/cassandra/data/cql_keyspace/table1-da46ad8043b711e6ad908f9393166cb4/mb-3-big,]
to level=0.
123.201MiB to 57.405MiB (~46% of original) in 44,527ms. Read Throughput =
2.767MiB/s, Write Throughput = 1.289MiB/s,
Row Throughput = ~444,444/s. 20,000,000 total partitions merged to
10,000,000. Partition merge counts were {2:1000, }
{quote}
*I'd like to start talking about compaction in terms of row throughput vs
MB/sec.*
There are some things we can do to make our current compaction faster like
CASSANDRA-10309 should give us a ~10% boost but the *bigger* issue and what I
think most users are seeing in the real world is the fact that under load
compaction effectively grinds to a halt. I can easily produce a 10x drop in
throughput by applying load during a regular compaction. This to me is the
problem we should be focused on since we can see this when looking at sample
user logs. I have a POC I'm working on to address this that I'll be presenting
soon to address this.
> Improve Compaction Throughput
> -
>
> Key: CASSANDRA-11697
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11697
> Project: Cassandra
> Issue Type: Improvement
> Components: Compaction
>Reporter: T Jake Luciani
>Assignee: T Jake Luciani
>
> The goal of this ticket is to improve/understand the bottlenecks during
> compactions. At a high level this will inv
