[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029577#comment-17029577
]
Pooja Nair edited comment on CASSANDRA-15529 at 2/4/20 4:47 AM:
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
We tried taking dump on multiple sstables but was thrown with similar
exception every time.
was (Author: pooja_nair):
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
We tried taking dump on multiple sstables but was thrown the similar exception
every time.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
> Attachments: sstable_dump.txt
>
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
>
[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029577#comment-17029577
]
Pooja Nair edited comment on CASSANDRA-15529 at 2/4/20 4:47 AM:
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
We tried taking dump on multiple sstables but was thrown the similar exception
every time.
was (Author: pooja_nair):
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
We tried taking dump on multiple sstables but was thrown the same exception
every time.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
> Attachments: sstable_dump.txt
>
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
>
[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029577#comment-17029577
]
Pooja Nair edited comment on CASSANDRA-15529 at 2/4/20 4:46 AM:
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
We tried taking dump on multiple sstables but was thrown the same exception
every time.
was (Author: pooja_nair):
Hi [~ifesdjeen]
I have attached the sstabledump of the table to this ticket.
We are getting an exception while trying to take a sstabledump for the above
mentioned sal_purge table where range deletes are carried out.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
> Attachments: sstable_dump.txt
>
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT bound as its last item at
> org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:2588)
> ~[apache-cassandra-3.11.4.jar:3.11.4] at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> [na:1.8.0-internal] at
>
[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029481#comment-17029481
]
feroz shaik edited comment on CASSANDRA-15529 at 2/4/20 2:35 AM:
-
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially impacting whole cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM
\"sal_purge\" USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
was (Author: ferozshaik...@gmail.com):
Hi [~ifesdjeen] , I am a colleague of [~Pooja_nair] and would like to answer
your questions. # We have all the nodes upgraded to 3.11.4 and also finished
upgradesstables. There are no previous sstables present on any of the
nodes/cluster.
# We have no clue whether the affected node is cordinator or not as it is too
random and the affected node behaves rogue potentially bring impacting whole
cluster.
# The query patterns are like below:
static String INSERT_SCHEDULED_DELETION_QUERY = "INSERT INTO \"sal_purge\"
(key,column1,value) VALUES (?,?,?) USING TIMESTAMP
?;"; static String SELECT_SCHEDULED_DELETION_QUERY = "SELECT column1, value
FROM sal_purge where key=? AND column1>=? LIMIT
?;"; static String DELETE_SCHEDULED_DELETION_QUERY = "DELETE FROM \"sal_purge\"
USING TIMESTAMP ? WHERE key=? AND column1=?;";
I will try to upload the sstable dump soon to this, Appreciate your support
[~ifesdjeen]
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
>
[jira] [Comment Edited] (CASSANDRA-15529) AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16 to 3.11.4
[
https://issues.apache.org/jira/browse/CASSANDRA-15529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17029059#comment-17029059
]
Alex Petrov edited comment on CASSANDRA-15529 at 2/3/20 4:04 PM:
-
[~Pooja_nair] is it possible to get a snapshot (possibly, anonymized if it's
produdciton data) of what sstable dump would look like? This looks like it
might be of a concern.
Does this happen on the upgraded SSTables or on the old ones? Do you have a
mixed cluster (as in, some nodes are still running old version)? If yes, does
it happen depending on which node is a coordinator or regardless?
It would be also good to know what kind of insert/update/delete query patterns
you roughly have for that schema, so I could try to reproduce locally.
was (Author: ifesdjeen):
[~Pooja_nair] is it possible to get a snapshot (possibly, anonymized if it's
produdciton data) of what sstable dump would look like? This looks like it
might be of a concern.
It would be also good to know what kind of insert/update/delete query patterns
you roughly have for that schema, so I could try to reproduce locally.
> AbstractLocalAwareExecutorService.java exceptions after upgrade from 2.1.16
> to 3.11.4
> -
>
> Key: CASSANDRA-15529
> URL: https://issues.apache.org/jira/browse/CASSANDRA-15529
> Project: Cassandra
> Issue Type: Bug
> Components: Cluster/Schema
>Reporter: Pooja Nair
>Priority: Urgent
> Labels: 2.1.16, 3.11.4
>
> Hello Team,
> We have cluster running on cassandra 3.11.4
> Following is the table schema of the tables that is being used in our system.
> {code:java}
> cqlsh> desc KEYSPACE "SAL"
>
> CREATE KEYSPACE "SAL" WITH replication = {'class':
> 'NetworkTopologyStrategy', 'DC_EAST': '3', 'DC_WEST': '3'} AND
> durable_writes = true;
>
> CREATE TABLE "SAL".sal_purge (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.1
> AND caching = '{"keys":"NONE", "rows_per_partition":"NONE"}'
> AND comment = 'Holds items to be removed as
> [shardid][salid][timestamp]. The table records SALIDs to be deleted along
> with their deletion times (which may be modified)'
> AND compaction = {'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.1
> AND speculative_retry = '99.0PERCENTILE';
>
> CREATE TABLE "SAL".sal_ref (
> key text,
> column1 text,
> column2 text,
> value text,
> PRIMARY KEY (key, column1, column2)
> ) WITH COMPACT STORAGE
> AND CLUSTERING ORDER BY (column1 ASC, column2 ASC)
> AND bloom_filter_fp_chance = 0.025
> AND caching = '{"keys":"ALL", "rows_per_partition":"NONE"}'
> AND comment = 'Holds owner references to content as [salid][lcid/opid]'
> AND compaction = {'sstable_size_in_mb': '180', 'class':
> 'org.apache.cassandra.db.compaction.LeveledCompactionStrategy'}
> AND compression = {'chunk_length_kb': '64', 'sstable_compression':
> 'org.apache.cassandra.io.compress.SnappyCompressor'}
> AND dclocal_read_repair_chance = 0.0
> AND default_time_to_live = 0
> AND gc_grace_seconds = 864000
> AND max_index_interval = 2048
> AND memtable_flush_period_in_ms = 0
> AND min_index_interval = 128
> AND read_repair_chance = 0.0
> AND speculative_retry = '99.0PERCENTILE';
> {code}
> Things to note:
> # The column2 is always passed a null value during insertion
> # column2 is a part of primary key
> # Range select and Range delete is done through our app.
> Iniatally the cluster was on casssandra version 2.1.16 and have been
> recently upgraded to 3.11.4 post the upgrade, we see that the nodes are going
> down, and log the below exceptions during startup and even after node is up.
> This one node is causing the whole cluster to behave improperly.
> {code:java}
> WARN [Native-Transport-Requests-47] 2020-01-29 13:49:05,190
> AbstractLocalAwareExecutorService.java:167 - Uncaught exception on thread
> Thread[Native-Transport-Requests-47,5,main]: {} java.lang.RuntimeException:
> java.lang.IllegalStateException: UnfilteredRowIterator for SAL.sal_purge has
> an open RT
