[
https://issues.apache.org/jira/browse/CASSANDRA-4314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13291009#comment-13291009
]
Jonathan Ellis commented on CASSANDRA-4314:
-------------------------------------------
bq. I have done no deletes
Then either you're using TTLs -- expired columns are basically the same as
deletes, in this respect -- or you're doing a lot of indexed column overwrites,
which also generate deletes in the index CF.
> OOM errors on key slice
> -----------------------
>
> Key: CASSANDRA-4314
> URL: https://issues.apache.org/jira/browse/CASSANDRA-4314
> Project: Cassandra
> Issue Type: Bug
> Components: Core
> Affects Versions: 1.1.0
> Environment: AS5 64, 64 GB ram, 12 core, Intel SSD
> Reporter: Wade Poziombka
> Attachments: oom.zip, yourkitsnapshot.png
>
>
> My database (now at 1.0.10) is in a state in which it goes out of memory with
> hardly any activity at all. A key slice nothing more.
> The logs attached are this including verbose gc in stdout. I started up
> cassandra and waited a bit to ensure that it was unperturbed.
> Then (about 15:46) I ran this slice (using Pelops), which in this case should
> return NO data. My client times out and the database goes OOM.
> ConsistencyLevel cl = ConsistencyLevel.TWO;//TWO nodes in
> my cluster
> Selector s = new Selector(this.pool);
> List<IndexExpression> indexExpressions = new
> ArrayList<IndexExpression>();
> IndexExpression e = new IndexExpression(
>
> ByteBuffer.wrap("encryptionSettingsID".getBytes(ASCII)), IndexOperator.EQ,
>
> ByteBuffer.wrap(encryptionSettingsID.getBytes(Utils.ASCII)));
> indexExpressions.add(e);
> IndexClause indexClause = new IndexClause(indexExpressions,
> ByteBuffer.wrap(EMPTY_BYTE_ARRAY), 1);
> SlicePredicate predicate = new SlicePredicate();
> predicate.setColumn_names(Arrays.asList(new ByteBuffer[]
> { ByteBuffer.wrap(COL_PAN_ENC_BYTES) }));
> List<KeySlice> slices = s.getKeySlices(CF_TOKEN,
> indexClause, predicate, cl);
> Note that “encryptionSettingsID” is an indexed column. When this is executed
> there should be no columns with the supplied value.
> I suppose I may have some kind of blatant error in this query but it is not
> obvious to me. I’m relatively new to cassandra.
> My key space is defined as follows:
> KsDef(name:TB_UNIT,
> strategy_class:org.apache.cassandra.locator.SimpleStrategy,
> strategy_options:{replication_factor=3},
> cf_defs:[
> CfDef(keyspace:TB_UNIT, name:token, column_type:Standard,
> comparator_type:BytesType, column_metadata:[ColumnDef(name:70 61 6E 45 6E 63,
> validation_class:BytesType), ColumnDef(name:63 72 65 61 74 65 54 73,
> validation_class:DateType), ColumnDef(name:63 72 65 61 74 65 44 61 74 65,
> validation_class:DateType, index_type:KEYS, index_name:TokenCreateDate),
> ColumnDef(name:65 6E 63 72 79 70 74 69 6F 6E 53 65 74 74 69 6E 67 73 49 44,
> validation_class:UTF8Type, index_type:KEYS,
> index_name:EncryptionSettingsID)], caching:keys_only),
> CfDef(keyspace:TB_UNIT, name:pan_d721fd40fd9443aa81cc6f59c8e047c6,
> column_type:Standard, comparator_type:BytesType, caching:keys_only),
> CfDef(keyspace:TB_UNIT, name:counters, column_type:Standard,
> comparator_type:BytesType, column_metadata:[ColumnDef(name:75 73 65 43 6F 75
> 6E 74, validation_class:CounterColumnType)],
> default_validation_class:CounterColumnType, caching:keys_only)
> ])
> tpstats show pending tasks many minutes after time out:
> [root@r610-lb6 bin]# ../cassandra/bin/nodetool -h 127.0.0.1 tpstats
> Pool Name Active Pending Completed Blocked All
> time blocked
> ReadStage 3 3 107 0
> 0
> RequestResponseStage 0 0 56 0
> 0
> MutationStage 0 0 6 0
> 0
> ReadRepairStage 0 0 0 0
> 0
> ReplicateOnWriteStage 0 0 0 0
> 0
> GossipStage 0 0 2231 0
> 0
> AntiEntropyStage 0 0 0 0
> 0
> MigrationStage 0 0 0 0
> 0
> MemtablePostFlusher 0 0 3 0
> 0
> StreamStage 0 0 0 0
> 0
> FlushWriter 0 0 3 0
> 0
> MiscStage 0 0 0 0
> 0
> InternalResponseStage 0 0 0 0
> 0
> HintedHandoff 0 0 9 0
> 0
> Message type Dropped
> RANGE_SLICE 0
> READ_REPAIR 0
> BINARY 0
> READ 0
> MUTATION 0
> REQUEST_RESPONSE 0
> cfstats:
> Keyspace: keyspace
> Read Count: 118
> Read Latency: 0.14722033898305084 ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Column Family: token
> SSTable count: 7
> Space used (live): 4745885584
> Space used (total): 4745885584
> Number of Keys (estimate): 18626048
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 118
> Read Latency: 0.147 ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 55058352
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 150
> Compacted row maximum size: 258
> Compacted row mean size: 201
> Column Family: pan_2fef6478b62242dd94aecaa049b9d7bb
> SSTable count: 7
> Space used (live): 1987147156
> Space used (total): 1987147156
> Number of Keys (estimate): 14955264
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 28056224
> Key cache: disabled
> Row cache: disabled
> Compacted row minimum size: 104
> Compacted row maximum size: 124
> Compacted row mean size: 124
> Column Family: counters
> SSTable count: 11
> Space used (live): 3433469364
> Space used (total): 3433469364
> Number of Keys (estimate): 21475328
> Memtable Columns Count: 0
> Memtable Data Size: 0
> Memtable Switch Count: 0
> Read Count: 0
> Read Latency: NaN ms.
> Write Count: 0
> Write Latency: NaN ms.
> Pending Tasks: 0
> Bloom Filter False Postives: 0
> Bloom Filter False Ratio: 0.00000
> Bloom Filter Space Used: 40271696
> Key cache capacity: 4652
> Key cache size: 4652
> Key cache hit rate: NaN
> Row cache: disabled
> Compacted row minimum size: 125
> Compacted row maximum size: 179
> Compacted row mean size: 150
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:
https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
