[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounta of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
>Reporter: Michael Penick
> Fix For: 1.2.12
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> co
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Environment:
dsc12-1.2.12-1.noarch.rpm
cassandra12-1.2.12-1.noarch.rpm
centos 6.4
was:
dsc12-1.2.12-1.noarch.rpm
cassandra12-1.2.12-1.noarch.rpm
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value even with much
> larger values.
> There are tons of these in the logs:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached are pictures of the GC under the different tests. Keep in mind we
> are only inserting 128KB - 256KB of data and we are almost hitting the limit
> of the heap.
> Example schemas:
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run a much faster insertion speed and much bigger
> column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column wit
[jira] [Commented] (CASSANDRA-6053) system.peers table not updated after decommissioning nodes in C* 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859609#comment-13859609 ] Tyler Hobbs commented on CASSANDRA-6053: I was able to repro with Ryan's steps, so I can upload the logs if you'd like, but I should be able to figure this one out. > system.peers table not updated after decommissioning nodes in C* 2.0 > > > Key: CASSANDRA-6053 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6053 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Datastax AMI running EC2 m1.xlarge instances >Reporter: Guyon Moree >Assignee: Tyler Hobbs > Attachments: peers > > > After decommissioning my cluster from 20 to 9 nodes using opscenter, I found > all but one of the nodes had incorrect system.peers tables. > This became a problem (afaik) when using the python-driver, since this > queries the peers table to set up its connection pool. Resulting in very slow > startup times, because of timeouts. > The output of nodetool didn't seem to be affected. After removing the > incorrect entries from the peers tables, the connection issues seem to have > disappeared for us. > Would like some feedback on if this was the right way to handle the issue or > if I'm still left with a broken cluster. > Attached is the output of nodetool status, which shows the correct 9 nodes. > Below that the output of the system.peers tables on the individual nodes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Attachment: GC_behavior.png
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value even with much
> larger values.
> There are tons of these in the logs:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached are pictures of the GC under the different tests. Keep in mind we
> are only inserting 128KB - 256KB of data and we are almost hitting the limit
> of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run a much faster insertion speed and much bigger
> column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under the one of the pathological tests. Keep
in mind we are only inserting 128KB - 256KB of data and we are almost hitting
the limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
>
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under one of the pathological tests. Keep in
mind we are only inserting 128KB - 256KB of data and we are almost hitting the
limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under the one of the pathological tests. Keep
in mind we are only inserting 128KB - 256KB of data and we are almost hitting
the limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
> Slow inserts with collections into a single partition (Pat
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under one of the pathological tests. Keep in
mind we are only inserting 128KB - 256KB of data and we are almost hitting the
limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run at a much faster insertion speed, for much
longer and much bigger column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under one of the pathological tests. Keep in
mind we are only inserting 128KB - 256KB of data and we are almost hitting the
limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
> Slow inserts with collections into a singl
[jira] [Updated] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Penick updated CASSANDRA-6534:
--
Description:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value and using much
larger values. During the collection insertion tests we have tons of these
messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under one of the pathological tests. Keep in
mind we are only inserting 128KB - 256KB of data and we are almost hitting the
limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison, I was able to run the same tests with the following schema
with no issue:
Note: This test was able to run at a much faster insertion speed, for much
longer and much bigger column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
was:
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values. There are also tons of these messages in the system.log:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounts of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached is a picture of the GC under one of the pathological tests. Keep in
mind we are only inserting 128KB - 256KB of data and we are almost hitting the
limit of the heap.
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Example schemas:
Note: The type of collection or primitive type in the collection doesn't seem
to matter.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run at a much faster insertion speed, for much
longer and much bigger column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{
[jira] [Commented] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859641#comment-13859641
]
Benedict commented on CASSANDRA-6534:
-
Are your test inserts being run in parallel, or serially? Are you overwriting
the map values with each insert, or adding new ones (if the latter, the
comparison between collections/non-collections is not fair)
There are a number of reasons why many rapid updates to a single partition key
may perform badly (this is not really the behaviour Cassandra is optimised
for), but I would try this out on the latest trunk to see if CASSANDRA-5417
helps you at all.
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859649#comment-13859649
]
Michael Penick commented on CASSANDRA-6534:
---
Sorry, That's a pretty big detail. The inserts are being run in parallel via
executeAsync() in java-driver. This is a wide row, so the collection values are
being inserted into a new column. So I don't think we're paying the cost to
deserialize and replace values in an existing map (if that's what you were
getting at?).
I just don't understand how the the collection serialization (or collection
processing) is making that big a difference compared to the non-colleciton
value. Wouldn't they both just end up as ByteBuffers in the storage engine?
The serialized size of the collections should be less than the largish values
we were inserting with the non-colleciton test. So where's all the extra memory
coming from?
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859649#comment-13859649
]
Michael Penick edited comment on CASSANDRA-6534 at 12/31/13 8:01 PM:
-
Sorry, That's a pretty big detail. The inserts are being run in parallel via
executeAsync() in java-driver. This is a wide row, so the collection values are
being inserted into a new column. So I don't think we're paying the cost to
deserialize and replace values in an existing map (if that's what you were
getting at?).
I just don't understand how the the collection serialization (or collection
processing) is making that big a difference compared to the non-colleciton
value. Wouldn't they both just end up as ByteBuffers in the storage engine?
The serialized size of the collections should be less than the largish values
we were inserting with the non-colleciton test. So where's all the extra
temporary heap memory coming from?
was (Author: mpenick):
Sorry, That's a pretty big detail. The inserts are being run in parallel via
executeAsync() in java-driver. This is a wide row, so the collection values are
being inserted into a new column. So I don't think we're paying the cost to
deserialize and replace values in an existing map (if that's what you were
getting at?).
I just don't understand how the the collection serialization (or collection
processing) is making that big a difference compared to the non-colleciton
value. Wouldn't they both just end up as ByteBuffers in the storage engine?
The serialized size of the collections should be less than the largish values
we were inserting with the non-colleciton test. So where's all the extra memory
coming from?
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859649#comment-13859649
]
Michael Penick edited comment on CASSANDRA-6534 at 12/31/13 8:03 PM:
-
Sorry, That's a pretty big detail. The inserts are being run in parallel via
executeAsync() in java-driver. This is a wide row, so the collection values are
always being inserted into a new column. So I don't think we're paying the cost
to deserialize and replace values in an existing map (if that's what you were
getting at?).
I just don't understand how the the collection serialization (or collection
processing) is making that big a difference compared to the non-colleciton
value. Wouldn't they both just end up as ByteBuffers in the storage engine?
The serialized size of the collections should be less than the largish values
we were inserting with the non-colleciton test. So where's all the extra
temporary heap memory coming from?
was (Author: mpenick):
Sorry, That's a pretty big detail. The inserts are being run in parallel via
executeAsync() in java-driver. This is a wide row, so the collection values are
being inserted into a new column. So I don't think we're paying the cost to
deserialize and replace values in an existing map (if that's what you were
getting at?).
I just don't understand how the the collection serialization (or collection
processing) is making that big a difference compared to the non-colleciton
value. Wouldn't they both just end up as ByteBuffers in the storage engine?
The serialized size of the collections should be less than the largish values
we were inserting with the non-colleciton test. So where's all the extra
temporary heap memory coming from?
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
[jira] [Commented] (CASSANDRA-6271) Replace SnapTree in AtomicSortedColumns
[ https://issues.apache.org/jira/browse/CASSANDRA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859658#comment-13859658 ] Jonathan Ellis commented on CASSANDRA-6271: --- bq. We can't re-use any sub-trees if we use parent pointers when modifying a btree, as the old btree needs to retain its prior parent pointers. Damn. bq. Personally I am in favour of the cleaned up stack version. Not mixing parameter-state with instance-state in succ/prev is a big improvement. > Replace SnapTree in AtomicSortedColumns > --- > > Key: CASSANDRA-6271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6271 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Labels: performance > Attachments: oprate.svg > > > On the write path a huge percentage of time is spent in GC (>50% in my tests, > if accounting for slow down due to parallel marking). SnapTrees are both GC > unfriendly due to their structure and also very expensive to keep around - > each column name in AtomicSortedColumns uses > 100 bytes on average > (excluding the actual ByteBuffer). > I suggest using a sorted array; changes are supplied at-once, as opposed to > one at a time, and if < 10% of the keys in the array change (and data equal > to < 10% of the size of the key array) we simply overlay a new array of > changes only over the top. Otherwise we rewrite the array. This method should > ensure much less GC overhead, and also save approximately 80% of the current > memory overhead. > TreeMap is similarly difficult object for the GC, and a related task might be > to remove it where not strictly necessary, even though we don't keep them > hanging around for long. TreeMapBackedSortedColumns, for instance, seems to > be used in a lot of places where we could simply sort the columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[
https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859661#comment-13859661
]
Eric Lubow commented on CASSANDRA-6494:
---
I am seeing this on a bootstrapping node on 1.2.11:
{quote}
ERROR 19:53:52,029 Exception in thread Thread[CompactionExecutor:93,1,main]
java.lang.RuntimeException: 01433f155800 is not defined as a collection
at
org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:69)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:81)
at
org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:31)
at
org.apache.cassandra.db.ArrayBackedSortedColumns.addColumn(ArrayBackedSortedColumns.java:128)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:119)
at
org.apache.cassandra.db.AbstractColumnContainer.addColumn(AbstractColumnContainer.java:114)
at org.apache.cassandra.db.ColumnFamily.addAtom(ColumnFamily.java:219)
at
org.apache.cassandra.db.ColumnFamilySerializer.deserializeColumnsFromSSTable(ColumnFamilySerializer.java:149)
at
org.apache.cassandra.io.sstable.SSTableIdentityIterator.getColumnFamilyWithColumns(SSTableIdentityIterator.java:234)
at
org.apache.cassandra.db.compaction.PrecompactedRow.merge(PrecompactedRow.java:114)
at
org.apache.cassandra.db.compaction.PrecompactedRow.(PrecompactedRow.java:98)
at
org.apache.cassandra.db.compaction.CompactionController.getCompactedRow(CompactionController.java:160)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:76)
at
org.apache.cassandra.db.compaction.CompactionIterable$Reducer.getReduced(CompactionIterable.java:57)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.consume(MergeIterator.java:114)
at
org.apache.cassandra.utils.MergeIterator$ManyToOne.computeNext(MergeIterator.java:97)
at
com.google.common.collect.AbstractIterator.tryToComputeNext(AbstractIterator.java:143)
at
com.google.common.collect.AbstractIterator.hasNext(AbstractIterator.java:138)
at
org.apache.cassandra.db.compaction.CompactionTask.runWith(CompactionTask.java:145)
at
org.apache.cassandra.io.util.DiskAwareRunnable.runMayThrow(DiskAwareRunnable.java:48)
at
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28)
at
org.apache.cassandra.db.compaction.CompactionTask.executeInternal(CompactionTask.java:58)
at
org.apache.cassandra.db.compaction.AbstractCompactionTask.execute(AbstractCompactionTask.java:60)
at
org.apache.cassandra.db.compaction.CompactionManager$BackgroundCompactionTask.run(CompactionManager.java:208)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:334)
at java.util.concurrent.FutureTask.run(FutureTask.java:166)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:724)
{quote}
> Cassandra refuses to restart due to a corrupted commit log.
> ---
>
> Key: CASSANDRA-6494
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6494
> Project: Cassandra
> Issue Type: Bug
>Reporter: Shao-Chuan Wang
>
> This is running on our production server. Please advise how to address this
> issue. Thank you!
> INFO 02:46:58,879 Finished reading
> /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log
> ERROR 02:46:58,879 Exception encountered during startup
> java.lang.RuntimeException: java.util.concurrent.ExecutionException:
> java.lang.RuntimeException: 706167655f74616773 is not defined as a collection
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411)
> at
> org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273)
> at
> org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146)
> at
> org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDae
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859665#comment-13859665 ] Jonathan Ellis commented on CASSANDRA-6494: --- That means it is trying to replay mutations from an invalid schema. What have you done to your schema recently? > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) >
[jira] [Comment Edited] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859672#comment-13859672 ] Eric Lubow edited comment on CASSANDRA-6494 at 12/31/13 8:26 PM: - We dropped a table named 'uniques' and recreated the same table named 'uniques' with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. was (Author: elubow): We dropped a table named 'uniques' and recreated the same table with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collect
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859672#comment-13859672 ] Eric Lubow commented on CASSANDRA-6494: --- We dropped a table named 'uniques' and recreated the same table with a different primary key. This error also appears to have frozen the bootstrap as nothing has happened on the bootstrap in over an hour. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > or
[jira] [Commented] (CASSANDRA-6271) Replace SnapTree in AtomicSortedColumns
[ https://issues.apache.org/jira/browse/CASSANDRA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859679#comment-13859679 ] Benedict commented on CASSANDRA-6271: - bq. Not mixing parameter-state with instance-state in succ/prev is a big improvement. Sometimes I get a little carried away with optimisation. > Replace SnapTree in AtomicSortedColumns > --- > > Key: CASSANDRA-6271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6271 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Labels: performance > Attachments: oprate.svg > > > On the write path a huge percentage of time is spent in GC (>50% in my tests, > if accounting for slow down due to parallel marking). SnapTrees are both GC > unfriendly due to their structure and also very expensive to keep around - > each column name in AtomicSortedColumns uses > 100 bytes on average > (excluding the actual ByteBuffer). > I suggest using a sorted array; changes are supplied at-once, as opposed to > one at a time, and if < 10% of the keys in the array change (and data equal > to < 10% of the size of the key array) we simply overlay a new array of > changes only over the top. Otherwise we rewrite the array. This method should > ensure much less GC overhead, and also save approximately 80% of the current > memory overhead. > TreeMap is similarly difficult object for the GC, and a related task might be > to remove it where not strictly necessary, even though we don't keep them > hanging around for long. TreeMapBackedSortedColumns, for instance, seems to > be used in a lot of places where we could simply sort the columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859690#comment-13859690
]
Benedict commented on CASSANDRA-6534:
-
bq. if that's what you were getting at?
No, actually the opposite. If you're storing new values every time in the same
wide row with composites/collections, you're going to be incurring large
overheads without the linked patch. The columns/keys have to be sorted, and the
comparison of two composite keys (which a collection key is backed by) requires
deserializing the components. Absolutely all of your inserts are also stored
against the same wide row, which has to be maintained in sorted order
(including all map elements) which means you have a single very large (100k+
elements) binary search tree in memory for the row.
However, if you're performing these in parallel it's quite possible the issue
is actually that the parallel modifications are all racing to complete, and
producing lots of aborted attempts to modify the partition key. Each partition
key is modified in a copy-on-write manner, and since the modifications are
potentially expensive due to the deserialization comparisons being performed
over the large binary search tree, they are highly likely to overlap, so each
insert is essentially performing a copy-on-write followed by an abort by all in
progress except the one successful operation. So you could be seeing a hugely
disproportionate generation of garbage, both in the tree modification and the
allocation of byte buffers that are discarded.
The linked patch might alleviate some of this by making the modifications
faster, so having fewer aborted races to modify, and some other improvements
that will reduce garbage are in the works that would marginally mitigate this
issue, but ultimately the underlying issue is that you should not be performing
large volumes of highly concurrent modifications to a single partition key.
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC
[jira] [Created] (CASSANDRA-6535) Prepared Statement on Defunct CF Can Impact Cluster Availability
Adam Holmberg created CASSANDRA-6535: Summary: Prepared Statement on Defunct CF Can Impact Cluster Availability Key: CASSANDRA-6535 URL: https://issues.apache.org/jira/browse/CASSANDRA-6535 Project: Cassandra Issue Type: Bug Components: Core Environment: Cassandra 1.2.12 CentOS 6.4 Reporter: Adam Holmberg *Synopsis:* misbehaving clients can cause DoS on a cluster with a defunct prepared statement *Scenario:* 1.) Create prepared INSERT statement on existing table X 2.) Table X is dropped 3.) Continue using prepared statement from (1) *Result:* a.) on coordinator node: COMMIT-LOG-WRITER + MutationStage errors b.) on other nodes: "UnknownColumnFamilyException reading from socket; closing" --> leads to thrashing inter-node connections c.) Other clients of the cluster suffer from I/O timeouts, presumably a result of (b) *Other observations:* * On single-node clusters, clients return from insert without error because mutation errors are swallowed. * On multiple-node clusters, clients receive a confounded 'read timeout' error because the closed internode connections do not propagate the error back. * With prepared SELECT statements (as opposed to INSERT described above). A NullPointerException is caused on the server, and no meaninful error is returned to the client. Besides the obvious "don't do that" to the integrator, it would be good if the cluster could handle this error case more gracefully and avoid undue impact. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Updated] (CASSANDRA-6535) Prepared Statement on Defunct CF Can Impact Cluster Availability
[ https://issues.apache.org/jira/browse/CASSANDRA-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Adam Holmberg updated CASSANDRA-6535: - Attachment: 6535.txt 6535.txt - a simple patch that adds CF validation to ClientState.hasColumnFamilyAccess. This buttons up the error pathology I was observing, preventing cluster impact and returning meaningful errors to the client. > Prepared Statement on Defunct CF Can Impact Cluster Availability > > > Key: CASSANDRA-6535 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6535 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 1.2.12 > CentOS 6.4 >Reporter: Adam Holmberg > Attachments: 6535.txt > > > *Synopsis:* misbehaving clients can cause DoS on a cluster with a defunct > prepared statement > *Scenario:* > 1.) Create prepared INSERT statement on existing table X > 2.) Table X is dropped > 3.) Continue using prepared statement from (1) > *Result:* > a.) on coordinator node: COMMIT-LOG-WRITER + MutationStage errors > b.) on other nodes: "UnknownColumnFamilyException reading from socket; > closing" --> leads to thrashing inter-node connections > c.) Other clients of the cluster suffer from I/O timeouts, presumably a > result of (b) > *Other observations:* > * On single-node clusters, clients return from insert without error because > mutation errors are swallowed. > * On multiple-node clusters, clients receive a confounded 'read timeout' > error because the closed internode connections do not propagate the error > back. > * With prepared SELECT statements (as opposed to INSERT described above). A > NullPointerException is caused on the server, and no meaninful error is > returned to the client. > Besides the obvious "don't do that" to the integrator, it would be good if > the cluster could handle this error case more gracefully and avoid undue > impact. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
[
https://issues.apache.org/jira/browse/CASSANDRA-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859709#comment-13859709
]
Michael Penick commented on CASSANDRA-6534:
---
I still confused why the type (collection vs non-colleciton) of the column
value would affect would affect the deserialization of the the column name. Are
both the column name and column value deserialized for the binary tree
insertion? I would have thought that the column name would have only been
needed to keep the sorted order.
Also, shouldn't both schemas (that share the exact same column name type)
result in approx. the same amount of contention regardless of the column value
type?
I'm just having trouble understanding why the type of the column value makes
any difference here. Again, I should note that the column name type is exactly
the same for both schemas.
> Slow inserts with collections into a single partition (Pathological GC
> behavior)
>
>
> Key: CASSANDRA-6534
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Environment: dsc12-1.2.12-1.noarch.rpm
> cassandra12-1.2.12-1.noarch.rpm
> centos 6.4
>Reporter: Michael Penick
> Fix For: 1.2.12
>
> Attachments: GC_behavior.png
>
>
> We noticed extremely slow insertion rates to a single partition key, using
> composite column with a collection value. We were not able to replicate the
> issue using the same schema, but with a non-colleciton value and using much
> larger values. During the collection insertion tests we have tons of these
> messages in the system.log:
> "GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
> collections, 1233256368 used; max is 8375238656"
> We are inserting a tiny amounts of data 32-64 bytes and seeing the issue
> after only a couple 10k inserts. The amount of memory being used by C*/JVM is
> no where near proportional to the amount data being inserted. Why is C*
> consuming so much memory?
> Attached is a picture of the GC under one of the pathological tests. Keep in
> mind we are only inserting 128KB - 256KB of data and we are almost hitting
> the limit of the heap.
> GC flags:
> -XX:+UseThreadPriorities
> -XX:ThreadPriorityPolicy=42
> -Xms8192M
> -Xmx8192M
> -Xmn2048M
> -XX:+HeapDumpOnOutOfMemoryError
> -Xss180k
> -XX:+UseParNewGC
> -XX:+UseConcMarkSweepGC
> -XX:+CMSParallelRemarkEnabled
> -XX:SurvivorRatio=8
> -XX:MaxTenuringThreshold=1
> -XX:CMSInitiatingOccupancyFraction=75
> -XX:+UseCMSInitiatingOccupancyOnly
> -XX:+UseTLAB
> Example schemas:
> Note: The type of collection or primitive type in the collection doesn't seem
> to matter.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value list,
> PRIMARY KEY(row_key, column_key));
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value map,
> PRIMARY KEY(row_key, column_key));
> {code}
> Example inserts:
> Note: This issue is able to be replicated with extremely small inserts (a
> well as larger ~1KB)
> {code}
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
> INSERT INTO test.test
> (row_key, column_key, column_value)
> VALUES
> ('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
> '0123456701234567012345670', 'b': '0123456701234567012345670' });
> {code}
> As a comparison, I was able to run the same tests with the following schema
> with no issue:
> Note: This test was able to run at a much faster insertion speed, for much
> longer and much bigger column sizes (1KB) without any GC issues.
> {code}
> CREATE TABLE test.test (
> row_key text,
> column_key uuid,
> column_value text,
> PRIMARY KEY(row_key, column_key) )
> {code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Resolved] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jonathan Ellis resolved CASSANDRA-6494. --- Resolution: Duplicate That would do it. CASSANDRA-5202 will fix this but until then you shouldn't drop/recreate with same name. If you don't need the data, removing the commitlog segment will allow starting up. If you do need it, you're looking at manual surgery on the commitlog to get rid of the pre-schema-change entries (e.g. by setting cfid to -1) which would allow the rest to go ahead. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.wait
[jira] [Comment Edited] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859722#comment-13859722 ] Jonathan Ellis edited comment on CASSANDRA-6494 at 12/31/13 9:44 PM: - That would do it. CASSANDRA-5202 will fix this but until then you shouldn't drop/recreate with same name. Looks like the drop didn't get all the sstables it needed to either (CASSANDRA-5911). You can probably identify the bad ones with scrub and manually nuke them. was (Author: jbellis): That would do it. CASSANDRA-5202 will fix this but until then you shouldn't drop/recreate with same name. If you don't need the data, removing the commitlog segment will allow starting up. If you do need it, you're looking at manual surgery on the commitlog to get rid of the pre-schema-change entries (e.g. by setting cfid to -1) which would allow the rest to go ahead. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run
[jira] [Updated] (CASSANDRA-6053) system.peers table not updated after decommissioning nodes in C* 2.0
[ https://issues.apache.org/jira/browse/CASSANDRA-6053?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tyler Hobbs updated CASSANDRA-6053: --- Attachment: 6053-v1.patch The problem was that state changes for the removed node were being handled after the system.peers row was deleted. These state changes would result in update to the system.peers row, partially reviving it. 6053-v1.patch (and [branch|https://github.com/thobbs/cassandra/tree/CASSANDRA-6053]) avoids updating the system.peers table if the node is unknown or is in one of the "dead" states and adds some basic unit test coverage. > system.peers table not updated after decommissioning nodes in C* 2.0 > > > Key: CASSANDRA-6053 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6053 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Datastax AMI running EC2 m1.xlarge instances >Reporter: Guyon Moree >Assignee: Tyler Hobbs > Attachments: 6053-v1.patch, peers > > > After decommissioning my cluster from 20 to 9 nodes using opscenter, I found > all but one of the nodes had incorrect system.peers tables. > This became a problem (afaik) when using the python-driver, since this > queries the peers table to set up its connection pool. Resulting in very slow > startup times, because of timeouts. > The output of nodetool didn't seem to be affected. After removing the > incorrect entries from the peers tables, the connection issues seem to have > disappeared for us. > Would like some feedback on if this was the right way to handle the issue or > if I'm still left with a broken cluster. > Attached is the output of nodetool status, which shows the correct 9 nodes. > Below that the output of the system.peers tables on the individual nodes. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
git commit: don't declare throwing exs that aren't thrown
Updated Branches:
refs/heads/trunk 49efc13cd -> 80548b359
don't declare throwing exs that aren't thrown
Project: http://git-wip-us.apache.org/repos/asf/cassandra/repo
Commit: http://git-wip-us.apache.org/repos/asf/cassandra/commit/80548b35
Tree: http://git-wip-us.apache.org/repos/asf/cassandra/tree/80548b35
Diff: http://git-wip-us.apache.org/repos/asf/cassandra/diff/80548b35
Branch: refs/heads/trunk
Commit: 80548b359ae254048145483b22e3fbe93fc90379
Parents: 49efc13
Author: Dave Brosius
Authored: Tue Dec 31 22:47:28 2013 -0500
Committer: Dave Brosius
Committed: Tue Dec 31 22:47:28 2013 -0500
--
src/java/org/apache/cassandra/cql3/statements/Restriction.java | 2 +-
1 file changed, 1 insertion(+), 1 deletion(-)
--
http://git-wip-us.apache.org/repos/asf/cassandra/blob/80548b35/src/java/org/apache/cassandra/cql3/statements/Restriction.java
--
diff --git a/src/java/org/apache/cassandra/cql3/statements/Restriction.java
b/src/java/org/apache/cassandra/cql3/statements/Restriction.java
index b6f900c..cadabf3 100644
--- a/src/java/org/apache/cassandra/cql3/statements/Restriction.java
+++ b/src/java/org/apache/cassandra/cql3/statements/Restriction.java
@@ -97,7 +97,7 @@ public interface Restriction
return new WithValues(values);
}
-public static IN create(Term value) throws InvalidRequestException
+public static IN create(Term value)
{
assert value instanceof Lists.Marker; // we shouldn't have got
there otherwise
return new WithMarker((Lists.Marker)value);
[jira] [Commented] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859833#comment-13859833 ] Shao-Chuan Wang commented on CASSANDRA-6494: We did drop the column and recreated the same column family with different schema. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cass
[jira] [Comment Edited] (CASSANDRA-6494) Cassandra refuses to restart due to a corrupted commit log.
[ https://issues.apache.org/jira/browse/CASSANDRA-6494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859833#comment-13859833 ] Shao-Chuan Wang edited comment on CASSANDRA-6494 at 1/1/14 5:27 AM: We did drop the column family and recreated the same column family with different schema. was (Author: shaochuan): We did drop the column and recreated the same column family with different schema. > Cassandra refuses to restart due to a corrupted commit log. > --- > > Key: CASSANDRA-6494 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6494 > Project: Cassandra > Issue Type: Bug >Reporter: Shao-Chuan Wang > > This is running on our production server. Please advise how to address this > issue. Thank you! > INFO 02:46:58,879 Finished reading > /mnt/cassandra/commitlog/CommitLog-3-1386069222785.log > ERROR 02:46:58,879 Exception encountered during startup > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:273) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.recover(CommitLogReplayer.java:96) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:146) > at > org.apache.cassandra.db.commitlog.CommitLog.recover(CommitLog.java:126) > at > org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:299) > at > org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:442) > at > org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:485) > Caused by: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at java.util.concurrent.FutureTask.report(FutureTask.java:122) > at java.util.concurrent.FutureTask.get(FutureTask.java:188) > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:407) > ... 8 more > Caused by: java.lang.RuntimeException: 706167655f74616773 is not defined as a > collection > at > org.apache.cassandra.db.marshal.ColumnToCollectionType.compareCollectionMembers(ColumnToCollectionType.java:72) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:85) > at > org.apache.cassandra.db.marshal.AbstractCompositeType.compare(AbstractCompositeType.java:35) > at > edu.stanford.ppl.concurrent.SnapTreeMap$1.compareTo(SnapTreeMap.java:538) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1108) > at > edu.stanford.ppl.concurrent.SnapTreeMap.attemptUpdate(SnapTreeMap.java:1192) > at > edu.stanford.ppl.concurrent.SnapTreeMap.updateUnderRoot(SnapTreeMap.java:1059) > at edu.stanford.ppl.concurrent.SnapTreeMap.update(SnapTreeMap.java:1023) > at > edu.stanford.ppl.concurrent.SnapTreeMap.putIfAbsent(SnapTreeMap.java:985) > at > org.apache.cassandra.db.AtomicSortedColumns$Holder.addColumn(AtomicSortedColumns.java:323) > at > org.apache.cassandra.db.AtomicSortedColumns.addAllWithSizeDelta(AtomicSortedColumns.java:195) > at org.apache.cassandra.db.Memtable.resolve(Memtable.java:196) > at org.apache.cassandra.db.Memtable.put(Memtable.java:160) > at > org.apache.cassandra.db.ColumnFamilyStore.apply(ColumnFamilyStore.java:842) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:373) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:338) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer$1.runMayThrow(CommitLogReplayer.java:265) > at > org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:28) > at > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:724) > java.lang.RuntimeException: java.util.concurrent.ExecutionException: > java.lang.RuntimeException: 706167655f74616773 is not defined as a collection > at > org.apache.cassandra.utils.FBUtilities.waitOnFuture(FBUtilities.java:411) > at > org.apache.cassandra.utils.FBUtilities.waitOnFutures(FBUtilities.java:400) > at > org.apache.cassandra.db.commitlog.CommitLogReplayer.r
[jira] [Commented] (CASSANDRA-6271) Replace SnapTree in AtomicSortedColumns
[ https://issues.apache.org/jira/browse/CASSANDRA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859515#comment-13859515 ] Benedict commented on CASSANDRA-6271: - I've uploaded an [updated version|https://github.com/belliottsmith/cassandra/tree/iss-6271] of the branch you're working on, with better separation of concerns for manipulating the stack in Path, and [another|https://github.com/belliottsmith/cassandra/tree/iss-6271-pp] with parent-pointers baked into the end of each node. Personally I am in favour of the cleaned up stack version. I think the clarity is similar for both in Cursor, but the code has become a little more polluted elsewhere to deal with parent pointers. Also, whilst I haven't benchmarked them, there is an easily perceptible drop in performance when running the testSlicingAllSmallTrees() test in the parent-pointer version. This will perhaps be a magnification of any real decline, though, as the tests run over a much smaller fan-factor. > Replace SnapTree in AtomicSortedColumns > --- > > Key: CASSANDRA-6271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6271 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Labels: performance > Attachments: oprate.svg > > > On the write path a huge percentage of time is spent in GC (>50% in my tests, > if accounting for slow down due to parallel marking). SnapTrees are both GC > unfriendly due to their structure and also very expensive to keep around - > each column name in AtomicSortedColumns uses > 100 bytes on average > (excluding the actual ByteBuffer). > I suggest using a sorted array; changes are supplied at-once, as opposed to > one at a time, and if < 10% of the keys in the array change (and data equal > to < 10% of the size of the key array) we simply overlay a new array of > changes only over the top. Otherwise we rewrite the array. This method should > ensure much less GC overhead, and also save approximately 80% of the current > memory overhead. > TreeMap is similarly difficult object for the GC, and a related task might be > to remove it where not strictly necessary, even though we don't keep them > hanging around for long. TreeMapBackedSortedColumns, for instance, seems to > be used in a lot of places where we could simply sort the columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Comment Edited] (CASSANDRA-6271) Replace SnapTree in AtomicSortedColumns
[ https://issues.apache.org/jira/browse/CASSANDRA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859515#comment-13859515 ] Benedict edited comment on CASSANDRA-6271 at 12/31/13 3:27 PM: --- I've uploaded an [updated version|https://github.com/belliottsmith/cassandra/tree/iss-6271] of the branch you're working on, with better separation of concerns for manipulating the stack in Path, and [another|https://github.com/belliottsmith/cassandra/tree/iss-6271-pp] with parent-pointers baked into the end of each node. Personally I am in favour of the cleaned up stack version. I think the clarity is similar for both in Cursor, but the code has become a little more polluted elsewhere to deal with parent pointers. Edit: The drop in performance was me messing with the thread count for the test, please ignore! Which makes more sense. was (Author: benedict): I've uploaded an [updated version|https://github.com/belliottsmith/cassandra/tree/iss-6271] of the branch you're working on, with better separation of concerns for manipulating the stack in Path, and [another|https://github.com/belliottsmith/cassandra/tree/iss-6271-pp] with parent-pointers baked into the end of each node. Personally I am in favour of the cleaned up stack version. I think the clarity is similar for both in Cursor, but the code has become a little more polluted elsewhere to deal with parent pointers. Also, whilst I haven't benchmarked them, there is an easily perceptible drop in performance when running the testSlicingAllSmallTrees() test in the parent-pointer version. This will perhaps be a magnification of any real decline, though, as the tests run over a much smaller fan-factor. > Replace SnapTree in AtomicSortedColumns > --- > > Key: CASSANDRA-6271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6271 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Labels: performance > Attachments: oprate.svg > > > On the write path a huge percentage of time is spent in GC (>50% in my tests, > if accounting for slow down due to parallel marking). SnapTrees are both GC > unfriendly due to their structure and also very expensive to keep around - > each column name in AtomicSortedColumns uses > 100 bytes on average > (excluding the actual ByteBuffer). > I suggest using a sorted array; changes are supplied at-once, as opposed to > one at a time, and if < 10% of the keys in the array change (and data equal > to < 10% of the size of the key array) we simply overlay a new array of > changes only over the top. Otherwise we rewrite the array. This method should > ensure much less GC overhead, and also save approximately 80% of the current > memory overhead. > TreeMap is similarly difficult object for the GC, and a related task might be > to remove it where not strictly necessary, even though we don't keep them > hanging around for long. TreeMapBackedSortedColumns, for instance, seems to > be used in a lot of places where we could simply sort the columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6271) Replace SnapTree in AtomicSortedColumns
[ https://issues.apache.org/jira/browse/CASSANDRA-6271?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859581#comment-13859581 ] Benedict commented on CASSANDRA-6271: - Ah, and I just realised I made a major mistake in my parent-pointer approach, which further damages the idea, probably a show stopper: We can't re-use *any* sub-trees if we use parent pointers when modifying a btree, as the old btree needs to retain its prior parent pointers. So a single column modification will need to rewrite the entire btree. We could reduce the impact of this by allocating wrapper objects that just contain the array and the parent pointer, but then we're getting into larger memory costs and more dereference costs when traversing. Also, further polluting the code that navigates with dealing with this extra level of indirection. Now that I think about it, I think this is why I ruled them out in the first place. > Replace SnapTree in AtomicSortedColumns > --- > > Key: CASSANDRA-6271 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6271 > Project: Cassandra > Issue Type: Improvement >Reporter: Benedict >Assignee: Benedict > Labels: performance > Attachments: oprate.svg > > > On the write path a huge percentage of time is spent in GC (>50% in my tests, > if accounting for slow down due to parallel marking). SnapTrees are both GC > unfriendly due to their structure and also very expensive to keep around - > each column name in AtomicSortedColumns uses > 100 bytes on average > (excluding the actual ByteBuffer). > I suggest using a sorted array; changes are supplied at-once, as opposed to > one at a time, and if < 10% of the keys in the array change (and data equal > to < 10% of the size of the key array) we simply overlay a new array of > changes only over the top. Otherwise we rewrite the array. This method should > ensure much less GC overhead, and also save approximately 80% of the current > memory overhead. > TreeMap is similarly difficult object for the GC, and a related task might be > to remove it where not strictly necessary, even though we don't keep them > hanging around for long. TreeMapBackedSortedColumns, for instance, seems to > be used in a lot of places where we could simply sort the columns. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Commented] (CASSANDRA-6245) "nodetool refresh" design is unsafe
[ https://issues.apache.org/jira/browse/CASSANDRA-6245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13859589#comment-13859589 ] Tyler Hobbs commented on CASSANDRA-6245: Since even I have been bitten by this in CASSANDRA-6514, I'm inclined to get this fixed. bq. Would you accept a patch which either : 1) moves "refresh" tasks so that they run from a subdirectory in the CF directory named "refresh" or 2) checks for file existence before creation, and inflates the sequence by one if the file exists [~rcoli] I think it's simpler and more sane to something like #1, although I would let it operate on an arbitrary directory (provided via a command line argument). A patch would be welcome. > "nodetool refresh" design is unsafe > --- > > Key: CASSANDRA-6245 > URL: https://issues.apache.org/jira/browse/CASSANDRA-6245 > Project: Cassandra > Issue Type: Bug > Components: Core >Reporter: Robert Coli >Priority: Minor > > CASSANDRA-2991 added a "nodetool refresh" feature by which Cassandra is able > to discover non-live SSTables in the datadir and make them live. > It does this by : > 1) looking for SSTable files in the data dir > 2) renaming SSTables it finds into the current SSTable id sequence > This implementation is exposed to a race with a chance of silent data loss. > 1) Node's SSTable id sequence is on sstable #2, the next table to flush will > get "2" as its numeric part > 2) Copy SSTable with "2" as its numeric part into data dir > 3) nodetool flush > 4) notice that your "2" SSTable has been silently overwritten by a > just-flushed "2" SSTable > 5) nodetool refresh would still succeed, but would now be a no-op > A simple solution would be to create a subdirectory of the datadir called > "refresh/" to serve as the location to refresh from. > Alternately/additionally, there is probably not really a compelling reason > for Cassandra to completely ignore existing files at write time.. a check for > existing files at a given index and inflating the index to avoid overwriting > them them seems trivial and inexpensive. I will gladly file a JIRA for this > change in isolation if there is interest. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (CASSANDRA-6478) Importing sstables through sstableloader tombstoned data
[
https://issues.apache.org/jira/browse/CASSANDRA-6478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tyler Hobbs resolved CASSANDRA-6478.
Resolution: Duplicate
Fix Version/s: (was: 2.0.5)
2.0.4
This appears to be an exact duplicate of CASSANDRA-6527 (which is resolved), so
I'm closing this ticket.
> Importing sstables through sstableloader tombstoned data
>
>
> Key: CASSANDRA-6478
> URL: https://issues.apache.org/jira/browse/CASSANDRA-6478
> Project: Cassandra
> Issue Type: Bug
> Components: Tools
> Environment: Cassandra 2.0.3
>Reporter: Mathijs Vogelzang
>Assignee: Tyler Hobbs
> Fix For: 2.0.4
>
>
> We've tried to import sstables from a snapshot of a 1.2.10 cluster into a
> running 2.0.3 cluster. When using sstableloader, for some reason we couldn't
> retrieve some of the data. When investigating further, it turned out that
> tombstones in the far future were created for some rows. (sstable2json
> returned the correct data, but with an addition of "metadata":
> {"deletionInfo":
> {"markedForDeleteAt":1796952039620607,"localDeletionTime":0}} to the rows
> that seemed missing).
> This happened again exactly the same way when we cleared the new cluster and
> ran sstableloader again.
> The sstables itself seemed fine, they were working on the old cluster,
> upgradesstables tells there's nothing to upgrade, and we were finally able to
> move our data correctly by copying the SSTables with scp into the right
> directory on the hosts of the new clusters worked fine (but naturally this
> required much more disk space than when sstableloader only sends the relevant
> parts).
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
[jira] [Created] (CASSANDRA-6534) Slow inserts with collections into a single partition (Pathological GC behavior)
Michael Penick created CASSANDRA-6534:
-
Summary: Slow inserts with collections into a single partition
(Pathological GC behavior)
Key: CASSANDRA-6534
URL: https://issues.apache.org/jira/browse/CASSANDRA-6534
Project: Cassandra
Issue Type: Bug
Components: Core
Environment: dsc12-1.2.12-1.noarch.rpm
cassandra12-1.2.12-1.noarch.rpm
GC flags:
-XX:+UseThreadPriorities
-XX:ThreadPriorityPolicy=42
-Xms8192M
-Xmx8192M
-Xmn2048M
-XX:+HeapDumpOnOutOfMemoryError
-Xss180k
-XX:+UseParNewGC
-XX:+UseConcMarkSweepGC
-XX:+CMSParallelRemarkEnabled
-XX:SurvivorRatio=8
-XX:MaxTenuringThreshold=1
-XX:CMSInitiatingOccupancyFraction=75
-XX:+UseCMSInitiatingOccupancyOnly
-XX:+UseTLAB
Reporter: Michael Penick
Fix For: 1.2.12
We noticed extremely slow insertion rates to a single partition key, using
composite column with a collection value. We were not able to replicate the
issue using the same schema, but with a non-colleciton value even with much
larger values.
There are tons of these in the logs:
"GCInspector.java (line 119) GC for ConcurrentMarkSweep: 1287 ms for 2
collections, 1233256368 used; max is 8375238656"
We are inserting a tiny amounta of data 32-64 bytes and seeing the issue after
only a couple 10k inserts. The amount of memory being used by C*/JVM is no
where near proportional to the amount data being inserted. Why is C* consuming
so much memory?
Attached are pictures of the GC under the different tests. Keep in mind we are
only inserting 128KB - 256KB of data and we are almost hitting the limit of the
heap.
Example schemas:
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value list,
PRIMARY KEY(row_key, column_key));
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value map,
PRIMARY KEY(row_key, column_key));
{code}
Example inserts:
Note: This issue is able to be replicated with extremely small inserts (a well
as larger ~1KB)
{code}
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('01', e0138677-7246-11e3-ac78-016ae7083d37, [0, 1, 2, 3]);
INSERT INTO test.test
(row_key, column_key, column_value)
VALUES
('22', 1ac5770a-7247-11e3-80e4-016ae7083d37, { 'a':
'0123456701234567012345670', 'b': '0123456701234567012345670' });
{code}
As a comparison I was able to run the same tests with the following schema with
no issue:
Note: This test was able to run a much faster insertion speed and much bigger
column sizes (1KB) without any GC issues.
{code}
CREATE TABLE test.test (
row_key text,
column_key uuid,
column_value text,
PRIMARY KEY(row_key, column_key) )
{code}
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
