[jira] [Created] (CASSANDRA-13795) DDL statements running slow on huge data
Vladimir created CASSANDRA-13795: Summary: DDL statements running slow on huge data Key: CASSANDRA-13795 URL: https://issues.apache.org/jira/browse/CASSANDRA-13795 Project: Cassandra Issue Type: Bug Reporter: Vladimir Fix For: 3.0.x We are facing the issues with Cassandra DDL statements with a huge amount of data inside. I run statement for create type and it works for several hours without success in cqlsh for remote server. -- This message was sent by Atlassian JIRA (v6.4.14#64029) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-9086) CAS precondition mismatch - row is still inserted
[
https://issues.apache.org/jira/browse/CASSANDRA-9086?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14644108#comment-14644108
]
Vladimir commented on CASSANDRA-9086:
-
Carl, could you please explain how in client code I can understood - was data
inserted by this last "failed" operation or it was exists before? Maybe
ResultSet have some additional information.
> CAS precondition mismatch - row is still inserted
> -
>
> Key: CASSANDRA-9086
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9086
> Project: Cassandra
> Issue Type: Bug
> Environment: Cassandra version 2.0.11
> 24 nodes in 4 datacenters (6 nodes per dc)
>Reporter: Roger Schildmeijer
>Assignee: Carl Yeksigian
>Priority: Critical
> Fix For: 2.0.15
>
> Attachments: conditional_insert_failure_cleaned.txt
>
>
> Cassandra claims CAS precondition didn't match current values[1], but the row
> is still inserted.
> We are having a quite simple cf schema:
> {code:xml}
> create table index (
> u text PRIMARY KEY,
> abc set,
> ) WITH
> compaction={'class': 'LeveledCompactionStrategy'};
> {code}
> CQL statement:
> {code:xml}
> INSERT INTO index(u, abc) VALUES(?, ?) IF NOT EXISTS
> {code}
> Sometimes, especially during some write load, Cassandra claims the
> conditional insert wasn't satisfied ([applied] false), but the row is still
> inserted.
> The response from the conditional insert contains [applied]=false + the data
> we tried to insert.
> A full CQL trace is attached.
> [1] CAS precondition CAS precondition
> org.apache.cassandra.cql3.statements.CQL3CasConditions@1b2576ce does not
> match current values ColumnFamily(username_index
> -{deletedAt=-9223372036854775808, localDeletion=2147483647,
> ranges=[abc-abc:!, deletedAt=1427889576525999, localDeletion=1427889564]}-
> [:false:0@1427889576526000,abc:39f1a470ee694761a8197c0d0ada8e8f:false:0@1427889576526000,])
> | 11:59:37.060 | /10.0.0.1 | 533702
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14519753#comment-14519753
]
Vladimir commented on CASSANDRA-9120:
-
I'd be happy to volunteer as developer for Checksum solution. We may start with
design initially. I think about that having separate file with checksums that
we could make it configurable and backward compatible with existing data. Let
me know if I can get in.
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.x, 2.1.x, 2.0.x
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518227#comment-14518227
]
Vladimir commented on CASSANDRA-9120:
-
Detect and report is fine. Can you please clarify, is there checksum in latest
versions already (like 3.0)?
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494867#comment-14494867
]
Vladimir commented on CASSANDRA-9120:
-
I mean JMX options, like "-Xmx maximum java heap size". If save that info
within the cache data, Cassandra could check that cache was saved when java
started with other options and skip it.
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494844#comment-14494844
]
Vladimir commented on CASSANDRA-9120:
-
It depends on what we are trying to fix:
- if corrupted file/cache I see no good solution w/o checksum.
- If operator reduced heap size or other - in this case we need to recognize it
and skip cache. In this case it could be good to add some information about
startup options (same file or separate).
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14494806#comment-14494806
]
Vladimir commented on CASSANDRA-9120:
-
If there is no way to add checksum, I'd try some simple: - create some status
file that indicates that we are trying to load cache file, delete it if loaded
successfully or failed by some reason that isn't OutOfMemory. Try to read that
file at next run and ignore cache. It's still fine to use
estimateMemorySizeForKeys as well but this isn't reliable as you mentioned and
can be useless.
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14485776#comment-14485776
]
Vladimir commented on CASSANDRA-9120:
-
Ariel, unfortunately I didn't save that corrupted cache. It's a good point that
operator can reduce heap size at hit this problem as well. Large-scale systems
are designed to avoid manual support at all. The changes rolls out to scale
unit, system monitors for errors and rolls out to next unit. Hard/soft re-image
is used to recover if unknown problem is detected.
Did I get it right that there are checksums already in latest version of
Cassandra?
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
>Assignee: Jeff Jirsa
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14396291#comment-14396291
]
Vladimir commented on CASSANDRA-9120:
-
What about saving checksums in a separate file? In thi case old versions will
use saved caved cache as is, new will check whole cache or separate records. Is
there reason to not have it? Otherwise it's still possible to read some
inconsistent data that wont cause a crash.
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
>Assignee: Jeff Jirsa
> Fix For: 3.0, 2.0.15, 2.1.5
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
[
https://issues.apache.org/jira/browse/CASSANDRA-9120?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14395833#comment-14395833
]
Vladimir commented on CASSANDRA-9120:
-
Our version 2.010 but this is a problem in latest too as far as I have seen.
> OutOfMemoryError when read auto-saved cache (probably broken)
> -
>
> Key: CASSANDRA-9120
> URL: https://issues.apache.org/jira/browse/CASSANDRA-9120
> Project: Cassandra
> Issue Type: Bug
> Environment: Linux
>Reporter: Vladimir
> Fix For: 2.0.15
>
>
> Found during tests on a 100 nodes cluster. After restart I found that one
> node constantly crashes with OutOfMemory Exception. I guess that auto-saved
> cache was corrupted and Cassandra can't recognize it. I see that similar
> issues was already fixed (when negative size of some structure was read).
> Does auto-saved cache have checksum? it'd help to reject corrupted cache at
> the very beginning.
> As far as I can see current code still have that problem. Stack trace is:
> {code}
> INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading
> saved cache
> /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db
> ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513)
> Exception encountered during startup
> java.lang.OutOfMemoryError: Java heap space
> at java.util.ArrayList.(Unknown Source)
> at
> org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120)
> at
> org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365)
> at
> org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119)
> at
> org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421)
> at
> org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392)
> at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315)
> at org.apache.cassandra.db.Keyspace.(Keyspace.java:272)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114)
> at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92)
> at
> org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536)
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585)
> {code}
> I looked at source code of Cassandra and see:
> http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java
> 119 int entries = in.readInt();
> 120 List columnsIndex = new
> ArrayList(entries);
> It seems that value entries is invalid (negative) and it tries too allocate
> an array with huge initial capacity and hits OOM. I have deleted saved_cache
> directory and was able to start node correctly. We should expect that it may
> happen in real world. Cassandra should be able to skip incorrect cached data
> and run.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (CASSANDRA-9120) OutOfMemoryError when read auto-saved cache (probably broken)
Vladimir created CASSANDRA-9120: --- Summary: OutOfMemoryError when read auto-saved cache (probably broken) Key: CASSANDRA-9120 URL: https://issues.apache.org/jira/browse/CASSANDRA-9120 Project: Cassandra Issue Type: Bug Environment: Linux Reporter: Vladimir Fix For: 2.0.14 Found during tests on a 100 nodes cluster. After restart I found that one node constantly crashes with OutOfMemory Exception. I guess that auto-saved cache was corrupted and Cassandra can't recognize it. I see that similar issues was already fixed (when negative size of some structure was read). Does auto-saved cache have checksum? it'd help to reject corrupted cache at the very beginning. As far as I can see current code still have that problem. Stack trace is: INFO [main] 2015-03-28 01:04:13,503 AutoSavingCache.java (line 114) reading saved cache /storage/core/loginsight/cidata/cassandra/saved_caches/system-sstable_activity-KeyCache-b.db ERROR [main] 2015-03-28 01:04:14,718 CassandraDaemon.java (line 513) Exception encountered during startup java.lang.OutOfMemoryError: Java heap space at java.util.ArrayList.(Unknown Source) at org.apache.cassandra.db.RowIndexEntry$Serializer.deserialize(RowIndexEntry.java:120) at org.apache.cassandra.service.CacheService$KeyCacheSerializer.deserialize(CacheService.java:365) at org.apache.cassandra.cache.AutoSavingCache.loadSaved(AutoSavingCache.java:119) at org.apache.cassandra.db.ColumnFamilyStore.(ColumnFamilyStore.java:262) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:421) at org.apache.cassandra.db.ColumnFamilyStore.createColumnFamilyStore(ColumnFamilyStore.java:392) at org.apache.cassandra.db.Keyspace.initCf(Keyspace.java:315) at org.apache.cassandra.db.Keyspace.(Keyspace.java:272) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:114) at org.apache.cassandra.db.Keyspace.open(Keyspace.java:92) at org.apache.cassandra.db.SystemKeyspace.checkHealth(SystemKeyspace.java:536) at org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:261) at org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:496) at org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:585) I looked at source code of Cassandra and see: http://grepcode.com/file/repo1.maven.org/maven2/org.apache.cassandra/cassandra-all/2.0.10/org/apache/cassandra/db/RowIndexEntry.java 119 int entries = in.readInt(); 120 List columnsIndex = new ArrayList(entries); It seems that value entries is invalid (negative) and it tries too allocate an array with huge initial capacity and hits OOM. I have deleted saved_cache directory and was able to start node correctly. We should expect that it may happen in real world. Cassandra should be able to skip incorrect cached data and run. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (CASSANDRA-9014) Cassandra log file has a leading space at every line
Vladimir created CASSANDRA-9014: --- Summary: Cassandra log file has a leading space at every line Key: CASSANDRA-9014 URL: https://issues.apache.org/jira/browse/CASSANDRA-9014 Project: Cassandra Issue Type: Bug Components: Core Environment: Linux Reporter: Vladimir Priority: Minor Fix For: 2.0.11 Every line at cassandra.log file has a leading space. Why? INFO [RepairJobTask:15] 2015-03-20 16:17:07,122 Differencer.java (line 67) [repair #87477a80-cf1c-11e4-9d1c-71a456073b65] Endpoints /10.153.204.21 and /10.153.204.2 are consistent for fields_by_id INFO [RepairJobTask:24] 2015-03-20 16:17:07,122 Differencer.java (line 67) [repair #87477a80-cf1c-11e4-9d1c-71a456073b65] Endpoints /10.153.204.20 and /10.153.204.2 are consistent for fields_by_id -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (CASSANDRA-3124) java heap limit for nodetool
[ https://issues.apache.org/jira/browse/CASSANDRA-3124?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14339292#comment-14339292 ] Vladimir commented on CASSANDRA-3124: - Today I tested 100 nodes cassandra cluster. `nodetool status` works but `nodetool ring' crashed with Out Of Memory: ./application/lib/apache-cassandra-2.0.10/bin/nodetool ring Note: Ownership information does not include topology; for complete information, specify a keyspace Datacenter: datacenter1 == Exception in thread "main" java.lang.OutOfMemoryError: Java heap space at java.util.Arrays.copyOf(Unknown Source) at java.util.ArrayList.grow(Unknown Source) at java.util.ArrayList.ensureExplicitCapacity(Unknown Source) at java.util.ArrayList.ensureCapacityInternal(Unknown Source) at java.util.ArrayList.addAll(Unknown Source) at org.apache.cassandra.tools.NodeCmd.printDc(NodeCmd.java:325) at org.apache.cassandra.tools.NodeCmd.printRing(NodeCmd.java:292) at org.apache.cassandra.tools.NodeCmd.main(NodeCmd.java:1224) I had to set up higher limit, 320MB at the script. Do you really think 32 Mb is enough or something wrong on my side? nodetool ring | wc -l 24842 ring.txt > java heap limit for nodetool > > > Key: CASSANDRA-3124 > URL: https://issues.apache.org/jira/browse/CASSANDRA-3124 > Project: Cassandra > Issue Type: Improvement > Components: Tools > Environment: not important >Reporter: Zenek Kraweznik >Assignee: Zenek Kraweznik >Priority: Minor > Fix For: 1.0.1 > > > by defaull (from debian package) > # nodetool > Error occurred during initialization of VM > Could not reserve enough space for object heap > Could not create the Java virtual machine. > # > and: > --- /usr/bin/nodetool.old 2011-09-02 14:15:14.228152799 +0200 > +++ /usr/bin/nodetool 2011-09-02 14:14:28.745154552 +0200 > @@ -55,7 +55,7 @@ > ;; > esac > -$JAVA -cp $CLASSPATH -Dstorage-config=$CASSANDRA_CONF \ > +$JAVA -Xmx32m -cp $CLASSPATH -Dstorage-config=$CASSANDRA_CONF \ > -Dlog4j.configuration=log4j-tools.properties \ > org.apache.cassandra.tools.NodeCmd $@ > after every upgrade i had to add limit manually. I think it's good idea to > add it by default ;) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
