[ https://issues.apache.org/jira/browse/CASSANDRA-8089?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14166698#comment-14166698 ]
Andrew S commented on CASSANDRA-8089: ------------------------------------- > nodetool cfstats Table: table_with_tombstone_issues SSTable count: 8 Space used (live), bytes: 33464296209 Space used (total), bytes: 33464296209 Space used by snapshots (total), bytes: 67910279993 SSTable Compression Ratio: 0.24045657357329253 Memtable cell count: 118368 Memtable data size, bytes: 9541504 Memtable switch count: 543 Local read count: 34239312 Local read latency: 0.092 ms Local write count: 580120 Local write latency: 0.086 ms Pending flushes: 0 Bloom filter false positives: 430017 Bloom filter false ratio: 0.98621 Bloom filter space used, bytes: 23072 Compacted partition minimum bytes: 216 Compacted partition maximum bytes: 30753941057 Compacted partition mean bytes: 6764845 Average live cells per slice (last five minutes): 0.3324931968307692 Average tombstones per slice (last five minutes): 0.6754286242686398 Table: table_unable_to_compute_ceiling_for_max_when_histogram_overflowed SSTable count: 11 Space used (live), bytes: 139688159520 Space used (total), bytes: 139688160740 Space used by snapshots (total), bytes: 53189649117 SSTable Compression Ratio: 0.23556415797023755 Memtable cell count: 439254 Memtable data size, bytes: 33330702 Memtable switch count: 817 Local read count: 0 Local read latency: NaN ms Local write count: 2692402 Local write latency: 0.081 ms Pending flushes: 0 Bloom filter false positives: 0 Bloom filter false ratio: 0.00000 Bloom filter space used, bytes: 3205776 Compacted partition minimum bytes: 373 Compacted partition maximum bytes: 110196931118 Compacted partition mean bytes: 144738 Average live cells per slice (last five minutes): 0.0 Average tombstones per slice (last five minutes): 0.0 > Invalid tombstone warnings / exceptions > --------------------------------------- > > Key: CASSANDRA-8089 > URL: https://issues.apache.org/jira/browse/CASSANDRA-8089 > Project: Cassandra > Issue Type: Bug > Components: Core > Environment: Cassandra 2.1.0 > Debian 7.6, 3.2.0-4-amd64 GNU/Linux > java version "1.7.0_51" > Java(TM) SE Runtime Environment (build 1.7.0_51-b13) > Java HotSpot(TM) 64-Bit Server VM (build 24.51-b03, mixed mode) > Reporter: Andrew S > > Hey, > We are having a strange issue with tombstone warnings which look like this: > {code} > WARN 12:28:42 Read 129 live and 4113 tombstoned cells in XXX.xxx (see > tombstone_warn_threshold). 500 columns was requested, > slices=[31660a4e-4f94-11e4-ac1d-53f244a29642-0a8073aa-4f9f-11e4-87c7-5b3e253389d8:!], > delInfo={deletedAt=-9223372036854775808, localDeletion=2147483647} > {code} > What is strange is that the row requested should not contain any tombstones > as we never delete data from that row. (We do delete data from other row in > the same column family) > To debug the issue we have dumped the data for this row using sstable2json > and the result does not contain any tombstones. (We have done this on all > nodes having the data and all sstables containing the key) > {code} > ./sstable2json /var/lib/cassandra/data/XXX/xxx/XXX-xxx-ka-81524-Data.db -k > "xxxxxxxxxxx" > {code} > We are getting the warnings after issuing a simple query: > {code} > select count(*) from xxx where key = 'xxxxx' and aggregate='xxxxx'; > {code} > There are only ~500 cells but it issues a warning about scanning 1700 > tombstones. > We are very worried about this because for some of the queries we are hitting > TombstoneOverwhelmingException for no obvious reason. > Here is the table definiion: > {code} > CREATE TABLE "Xxxx".xxx ( > key text, > aggregate text, > t timeuuid, > ..... {date fields } > PRIMARY KEY ((key, aggregate), t) > ) WITH CLUSTERING ORDER BY (t ASC) > AND bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = 'we love cassandra' > AND compaction = {'min_threshold': '6', 'class': > 'org.apache.cassandra.db.compaction.SizeTieredCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.SnappyCompressor'} > AND dclocal_read_repair_chance = 0.0 > AND default_time_to_live = 0 > AND gc_grace_seconds = 3600 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.1 > AND speculative_retry = '99.0PERCENTILE'; > {code} > Do you have any ideas how can we debug this further? > Thanks, > Andrew -- This message was sent by Atlassian JIRA (v6.3.4#6332)