[ https://issues.apache.org/jira/browse/CASSANDRA-10137?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sergey updated CASSANDRA-10137: ------------------------------- Description: I have 2 dc and 3 node: dc1: 2 node; dc2: 1 node; Exist keyspace KEYSPACE itm_dhcp_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true; and CF: TABLE itm_dhcp_test.lock ( name text PRIMARY KEY, reason text, time timestamp, who text ) WITH bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; Periodically there is a problem with deleting records. For example execute query: INSERT INTO lock (name, reason, time, who) values ('unitTest4', 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60 SELECT * FROM lock WHERE name='unitTest4' DELETE FROM lock WHERE name='unitTest4' SELECT * FROM lock WHERE name='unitTest4' 20% - 30% of cases last SELECT returns not empty record. Most often when coordinator node1-dc2. In trace I see the message: | Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | 45 | SharedPool-Worker-3 | Preparing statement | node1.dc2 | 151 | SharedPool-Worker-3 | Executing single-partition query on users | node1.dc2 | 588 | SharedPool-Worker-1 | Acquiring sstable references | node1.dc2 | 601 | SharedPool-Worker-1 | Merging memtable tombstones | node1.dc2 | 634 | SharedPool-Worker-1 | Key cache hit for sstable 2 | node1.dc2| 668 | SharedPool-Worker-1 | Seeking to partition beginning in data file | node1.dc2 | 674 | SharedPool-Worker-1 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 737 | SharedPool-Worker-1 | Merging data from memtables and 1 sstables | node1.dc2 | 743 | SharedPool-Worker-1 | Read 1 live and 0 tombstoned cells | node1.dc2 | 795 | SharedPool-Worker-1 | Executing single-partition query on permissions | node1.dc2 | 1653 | SharedPool-Worker-1 | Acquiring sstable references | node1.dc2 | 1662 | SharedPool-Worker-1 | Merging memtable tombstones | node1.dc2 | 1690 | SharedPool-Worker-1 | Key cache hit for sstable 5 | node1.dc2| 1737 | SharedPool-Worker-1 | Seeking to partition indexed section in data file | node1.dc2 | 1742 | SharedPool-Worker-1 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 1797 | SharedPool-Worker-1 | Merging data from memtables and 1 sstables | node1.dc2 | 1805 | SharedPool-Worker-1 | Read 0 live and 0 tombstoned cells | node1.dc2 | 1819 | SharedPool-Worker-1 | Executing single-partition query on users | node1.dc2 | 2798 | SharedPool-Worker-4 | Acquiring sstable references | node1.dc2 | 2808 | SharedPool-Worker-4 | Merging memtable tombstones | node1.dc2 | 2851 | SharedPool-Worker-4 | Key cache hit for sstable 2 | node1.dc2| 2896 | SharedPool-Worker-4 | Seeking to partition beginning in data file | node1.dc2 | 2903 | SharedPool-Worker-4 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 2948 | SharedPool-Worker-4 | Merging data from memtables and 1 sstables | node1.dc2 | 2954 | SharedPool-Worker-4 | Read 1 live and 0 tombstoned cells | node1.dc2 | 3004 | SharedPool-Worker-4 | Executing single-partition query on permissions | node1.dc2 | 3807 | SharedPool-Worker-4 | Acquiring sstable references | node1.dc2 | 3820 | SharedPool-Worker-4 | Merging memtable tombstones | node1.dc2 | 3861 | SharedPool-Worker-4 | Key cache hit for sstable 5 | node1.dc2| 3927 | SharedPool-Worker-4 | Seeking to partition indexed section in data file | node1.dc2 | 3933 | SharedPool-Worker-4 | Please help understand, why the record is not deleted? was: I have 2 dc and 3 node: dc1: 2 node; dc2: 1 node; Exist keyspace KEYSPACE itm_dhcp_test WITH replication = {'class': 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true; and CF: TABLE itm_dhcp_test.lock ( name text PRIMARY KEY, reason text, time timestamp, who text ) WITH bloom_filter_fp_chance = 0.01 AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' AND comment = '' AND compaction = {'min_threshold': '4', 'class': 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', 'max_threshold': '32'} AND compression = {'sstable_compression': 'org.apache.cassandra.io.compress.LZ4Compressor'} AND dclocal_read_repair_chance = 0.1 AND default_time_to_live = 0 AND gc_grace_seconds = 864000 AND max_index_interval = 2048 AND memtable_flush_period_in_ms = 0 AND min_index_interval = 128 AND read_repair_chance = 0.0 AND speculative_retry = '99.0PERCENTILE'; Periodically there is a problem with deleting records. For example execute query: INSERT INTO lock (name, reason, time, who) values ('unitTest4', 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60 SELECT * FROM lock WHERE name='unitTest4' DELETE FROM lock WHERE name='unitTest4' SELECT * FROM lock WHERE name='unitTest4' 20% - 30% of cases last SELECT returns not empty record. Most often when coordinator node1-dc2. In trace I see the message: Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | 45 | SharedPool-Worker-3 | Preparing statement | node1.dc2 | 151 | SharedPool-Worker-3 | Executing single-partition query on users | node1.dc2 | 588 | SharedPool-Worker-1 | Acquiring sstable references | node1.dc2 | 601 | SharedPool-Worker-1 | Merging memtable tombstones | node1.dc2 | 634 | SharedPool-Worker-1 | Key cache hit for sstable 2 | node1.dc2| 668 | SharedPool-Worker-1 | Seeking to partition beginning in data file | node1.dc2 | 674 | SharedPool-Worker-1 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 737 | SharedPool-Worker-1 | Merging data from memtables and 1 sstables | node1.dc2 | 743 | SharedPool-Worker-1 | Read 1 live and 0 tombstoned cells | node1.dc2 | 795 | SharedPool-Worker-1 | Executing single-partition query on permissions | node1.dc2 | 1653 | SharedPool-Worker-1 | Acquiring sstable references | node1.dc2 | 1662 | SharedPool-Worker-1 | Merging memtable tombstones | node1.dc2 | 1690 | SharedPool-Worker-1 | Key cache hit for sstable 5 | node1.dc2| 1737 | SharedPool-Worker-1 | Seeking to partition indexed section in data file | node1.dc2 | 1742 | SharedPool-Worker-1 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 1797 | SharedPool-Worker-1 | Merging data from memtables and 1 sstables | node1.dc2 | 1805 | SharedPool-Worker-1 | Read 0 live and 0 tombstoned cells | node1.dc2 | 1819 | SharedPool-Worker-1 | Executing single-partition query on users | node1.dc2 | 2798 | SharedPool-Worker-4 | Acquiring sstable references | node1.dc2 | 2808 | SharedPool-Worker-4 | Merging memtable tombstones | node1.dc2 | 2851 | SharedPool-Worker-4 | Key cache hit for sstable 2 | node1.dc2| 2896 | SharedPool-Worker-4 | Seeking to partition beginning in data file | node1.dc2 | 2903 | SharedPool-Worker-4 | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | node1.dc2 | 2948 | SharedPool-Worker-4 | Merging data from memtables and 1 sstables | node1.dc2 | 2954 | SharedPool-Worker-4 | Read 1 live and 0 tombstoned cells | node1.dc2 | 3004 | SharedPool-Worker-4 | Executing single-partition query on permissions | node1.dc2 | 3807 | SharedPool-Worker-4 | Acquiring sstable references | node1.dc2 | 3820 | SharedPool-Worker-4 | Merging memtable tombstones | node1.dc2 | 3861 | SharedPool-Worker-4 | Key cache hit for sstable 5 | node1.dc2| 3927 | SharedPool-Worker-4 | Seeking to partition indexed section in data file | node1.dc2 | 3933 | SharedPool-Worker-4 Please help understand, why the record is not deleted? > Consistency problem > ------------------- > > Key: CASSANDRA-10137 > URL: https://issues.apache.org/jira/browse/CASSANDRA-10137 > Project: Cassandra > Issue Type: Bug > Reporter: Sergey > > I have 2 dc and 3 node: > dc1: 2 node; > dc2: 1 node; > Exist keyspace > KEYSPACE itm_dhcp_test WITH replication = {'class': > 'NetworkTopologyStrategy', 'DC1': '2', 'DC2': '1'} AND durable_writes = true; > and CF: > TABLE itm_dhcp_test.lock ( > name text PRIMARY KEY, > reason text, > time timestamp, > who text > ) WITH bloom_filter_fp_chance = 0.01 > AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}' > AND comment = '' > AND compaction = {'min_threshold': '4', 'class': > 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy', > 'max_threshold': '32'} > AND compression = {'sstable_compression': > 'org.apache.cassandra.io.compress.LZ4Compressor'} > AND dclocal_read_repair_chance = 0.1 > AND default_time_to_live = 0 > AND gc_grace_seconds = 864000 > AND max_index_interval = 2048 > AND memtable_flush_period_in_ms = 0 > AND min_index_interval = 128 > AND read_repair_chance = 0.0 > AND speculative_retry = '99.0PERCENTILE'; > Periodically there is a problem with deleting records. For example execute > query: > INSERT INTO lock (name, reason, time, who) values ('unitTest4', > 'CassandraClusterLockTest', dateof(now()), 'I') IF NOT EXISTS USING TTL 60 > SELECT * FROM lock WHERE name='unitTest4' > DELETE FROM lock WHERE name='unitTest4' > SELECT * FROM lock WHERE name='unitTest4' > 20% - 30% of cases last SELECT returns not empty record. Most often when > coordinator node1-dc2. > In trace I see the message: > | Parsing DELETE FROM lock WHERE name='unitTest4' | node1.dc2 | > 45 | SharedPool-Worker-3 > | Preparing statement | > node1.dc2 | 151 | SharedPool-Worker-3 > | Executing single-partition query on users | > node1.dc2 | 588 | SharedPool-Worker-1 > | Acquiring sstable references | > node1.dc2 | 601 | SharedPool-Worker-1 > | Merging memtable tombstones | > node1.dc2 | 634 | SharedPool-Worker-1 > | Key cache hit for sstable 2 | > node1.dc2| 668 | SharedPool-Worker-1 > | Seeking to partition beginning in data file | > node1.dc2 | 674 | SharedPool-Worker-1 > | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | > node1.dc2 | 737 | SharedPool-Worker-1 > | Merging data from memtables and 1 sstables | > node1.dc2 | 743 | SharedPool-Worker-1 > | Read 1 live and 0 tombstoned cells | > node1.dc2 | 795 | SharedPool-Worker-1 > | Executing single-partition query on permissions | > node1.dc2 | 1653 | SharedPool-Worker-1 > | Acquiring sstable references | > node1.dc2 | 1662 | SharedPool-Worker-1 > | Merging memtable tombstones | > node1.dc2 | 1690 | SharedPool-Worker-1 > | Key cache hit for sstable 5 | > node1.dc2| 1737 | SharedPool-Worker-1 > | Seeking to partition indexed section in data file | > node1.dc2 | 1742 | SharedPool-Worker-1 > | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | > node1.dc2 | 1797 | SharedPool-Worker-1 > | Merging data from memtables and 1 sstables | > node1.dc2 | 1805 | SharedPool-Worker-1 > | Read 0 live and 0 tombstoned cells | > node1.dc2 | 1819 | SharedPool-Worker-1 > | Executing single-partition query on users | > node1.dc2 | 2798 | SharedPool-Worker-4 > | Acquiring sstable references | > node1.dc2 | 2808 | SharedPool-Worker-4 > | Merging memtable tombstones | > node1.dc2 | 2851 | SharedPool-Worker-4 > | Key cache hit for sstable 2 | > node1.dc2| 2896 | SharedPool-Worker-4 > | Seeking to partition beginning in data file | > node1.dc2 | 2903 | SharedPool-Worker-4 > | Skipped 0/1 non-slice-intersecting sstables, included 0 due to tombstones | > node1.dc2 | 2948 | SharedPool-Worker-4 > | Merging data from memtables and 1 sstables | > node1.dc2 | 2954 | SharedPool-Worker-4 > | Read 1 live and 0 tombstoned cells | > node1.dc2 | 3004 | SharedPool-Worker-4 > | Executing single-partition query on permissions | > node1.dc2 | 3807 | SharedPool-Worker-4 > | Acquiring sstable references | > node1.dc2 | 3820 | SharedPool-Worker-4 > | Merging memtable tombstones | > node1.dc2 | 3861 | SharedPool-Worker-4 > | Key cache hit for sstable 5 | > node1.dc2| 3927 | SharedPool-Worker-4 > | Seeking to partition indexed section in data file | > node1.dc2 | 3933 | SharedPool-Worker-4 > | > Please help understand, why the record is not deleted? -- This message was sent by Atlassian JIRA (v6.3.4#6332)