[ https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Penick updated CASSANDRA-6534: -------------------------------------- Description: We noticed extremely slow insertion rates to a single partition key, using composite column with a collection value. We were not able to replicate the issue using the same schema, but with a non-colleciton value even with much larger values. There are tons of these in the logs: "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2 collections, 1233256368 used; max is 8375238656" We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after only a couple 10k inserts. The amount of memory being used by C*/JVM is no where near proportional to the amount data being inserted. Why is C* consuming so much memory? Attached are pictures of the GC under the different tests. Keep in mind we are only inserting 128KB - 256KB of data and we are almost hitting the limit of the heap. GC flags: -XX:+UseThreadPriorities -XX:ThreadPriorityPolicy=42 -Xms8192M -Xmx8192M -Xmn2048M -XX:+HeapDumpOnOutOfMemoryError -Xss180k -XX:+UseParNewGC -XX:+UseConcMarkSweepGC -XX:+CMSParallelRemarkEnabled -XX:SurvivorRatio=8 -XX:MaxTenuringThreshold=1 -XX:CMSInitiatingOccupancyFraction=75 -XX:+UseCMSInitiatingOccupancyOnly -XX:+UseTLAB Example schemas: {code} CREATE TABLE test.test ( row_key text, column_key uuid, column_value list<int>, PRIMARY KEY(row_key, column_key)); CREATE TABLE test.test ( row_key text, column_key uuid, column_value map<text, text>, PRIMARY KEY(row_key, column_key)); {code} Example inserts: Note: This issue is able to be replicated with extremely small inserts (a well as larger ~1KB) {code} INSERT INTO test.test (row_key, column_key, column_value) VALUES ('0000000001', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]); INSERT INTO test.test (row_key, column_key, column_value) VALUES ('0000000022', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a': '0123456701234567012345670', 'b': '0123456701234567012345670' }); {code} As a comparison I was able to run the same tests with the following schema with no issue: Note: This test was able to run a much faster insertion speed and much bigger column sizes (1KB) without any GC issues. {code} CREATE TABLE test.test ( row_key text, column_key uuid, column_value text, PRIMARY KEY(row_key, column_key) ) {code} was: We noticed extremely slow insertion rates to a single partition key, using composite column with a collection value. We were not able to replicate the issue using the same schema, but with a non-colleciton value even with much larger values. There are tons of these in the logs: "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2 collections, 1233256368 used; max is 8375238656" We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after only a couple 10k inserts. The amount of memory being used by C*/JVM is no where near proportional to the amount data being inserted. Why is C* consuming so much memory? Attached are pictures of the GC under the different tests. Keep in mind we are only inserting 128KB - 256KB of data and we are almost hitting the limit of the heap. Example schemas: {code} CREATE TABLE test.test ( row_key text, column_key uuid, column_value list<int>, PRIMARY KEY(row_key, column_key)); CREATE TABLE test.test ( row_key text, column_key uuid, column_value map<text, text>, PRIMARY KEY(row_key, column_key)); {code} Example inserts: Note: This issue is able to be replicated with extremely small inserts (a well as larger ~1KB) {code} INSERT INTO test.test (row_key, column_key, column_value) VALUES ('0000000001', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]); INSERT INTO test.test (row_key, column_key, column_value) VALUES ('0000000022', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a': '0123456701234567012345670', 'b': '0123456701234567012345670' }); {code} As a comparison I was able to run the same tests with the following schema with no issue: Note: This test was able to run a much faster insertion speed and much bigger column sizes (1KB) without any GC issues. {code} CREATE TABLE test.test ( row_key text, column_key uuid, column_value text, PRIMARY KEY(row_key, column_key) ) {code} > Slow inserts with collections into a single partition (Pathological GC > behavior) > -------------------------------------------------------------------------------- > > Key: CASSANDRA-6534 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6534 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: dsc12-1.2.12-1.noarch.rpm > cassandra12-1.2.12-1.noarch.rpm > centos 6.4 > Reporter: Michael Penick > Fix For: 1.2.12 > > > We noticed extremely slow insertion rates to a single partition key, using > composite column with a collection value. We were not able to replicate the > issue using the same schema, but with a non-colleciton value even with much > larger values. > There are tons of these in the logs: > "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2 > collections, 1233256368 used; max is 8375238656" > We are inserting a tiny amounts of data 32-64 bytes and seeing the issue > after only a couple 10k inserts. The amount of memory being used by C*/JVM is > no where near proportional to the amount data being inserted. Why is C* > consuming so much memory? > Attached are pictures of the GC under the different tests. Keep in mind we > are only inserting 128KB - 256KB of data and we are almost hitting the limit > of the heap. > GC flags: > -XX:+UseThreadPriorities > -XX:ThreadPriorityPolicy=42 > -Xms8192M > -Xmx8192M > -Xmn2048M > -XX:+HeapDumpOnOutOfMemoryError > -Xss180k > -XX:+UseParNewGC > -XX:+UseConcMarkSweepGC > -XX:+CMSParallelRemarkEnabled > -XX:SurvivorRatio=8 > -XX:MaxTenuringThreshold=1 > -XX:CMSInitiatingOccupancyFraction=75 > -XX:+UseCMSInitiatingOccupancyOnly > -XX:+UseTLAB > Example schemas: > {code} > CREATE TABLE test.test ( > row_key text, > column_key uuid, > column_value list<int>, > PRIMARY KEY(row_key, column_key)); > CREATE TABLE test.test ( > row_key text, > column_key uuid, > column_value map<text, text>, > PRIMARY KEY(row_key, column_key)); > {code} > Example inserts: > Note: This issue is able to be replicated with extremely small inserts (a > well as larger ~1KB) > {code} > INSERT INTO test.test > (row_key, column_key, column_value) > VALUES > ('0000000001', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]); > INSERT INTO test.test > (row_key, column_key, column_value) > VALUES > ('0000000022', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a': > '0123456701234567012345670', 'b': '0123456701234567012345670' }); > {code} > As a comparison I was able to run the same tests with the following schema > with no issue: > Note: This test was able to run a much faster insertion speed and much bigger > column sizes (1KB) without any GC issues. > {code} > CREATE TABLE test.test ( > row_key text, > column_key uuid, > column_value text, > PRIMARY KEY(row_key, column_key) ) > {code} -- This message was sent by Atlassian JIRA (v6.1.5#6160)