[
https://issues.apache.org/jira/browse/CASSANDRA-20725?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Christoph Schnepf updated CASSANDRA-20725:
------------------------------------------
Description:
Hi,
we did the update from Cassandra 4.1.8 to Cassandra 4.1.9 on a 30-node cluster
in a rolling fashion, however, on 1 node, 1 single SSTable was showing a
NullPointerException.
{code:java}
INFO [main] 2025-06-23 10:41:38,576 UTC ColumnFamilyStore.java:484 -
Initializing mykeyspace.MyTWCSTable
INFO [SSTableBatchOpen:2] 2025-06-23 10:41:38,583 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1008071-big
(8.141GiB)
...
... more "Opening" lines
...
INFO [SSTableBatchOpen:32] 2025-06-23 10:41:38,586 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-997878-big
(7.357GiB)
ERROR [SSTableBatchOpen:27] 2025-06-23 10:41:38,586 UTC
JVMStabilityInspector.java:68 - Exception in thread
Thread[SSTableBatchOpen:27,5,SSTableBatchOpen]
java.lang.NullPointerException: null
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:510)
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:381)
at
org.apache.cassandra.io.sstable.format.SSTableReader$2.run(SSTableReader.java:551)
at
org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
INFO [SSTableBatchOpen:13] 2025-06-23 10:41:38,594 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1013913-big
(307.642MiB)
INFO [SSTableBatchOpen:27] 2025-06-23 10:41:38,629 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1001329-big
(7.481GiB)
INFO [SSTableBatchOpen:26] 2025-06-23 10:41:38,679 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1005032-big
(8.213GiB)
DEBUG [main] 2025-06-23 10:41:39,228 UTC DiskBoundaryManager.java:54 -
Refreshing disk boundary cache for mykeyspace.MyTWCSTable
DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:93 - Got
local ranges [<removed-for-readability>] (ringVersion = 30)
DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:57 - Updating
boundaries from null to
DiskBoundaries{directories=[DataDirectory{location=/var/opt/cassandra}],
positions=[max(9223372036854775807)], ringVersion=30, directoriesVersion=0} for
mykeyspace.MyTWCSTable
DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68 -
Enabling tombstone compactions for TWCS
DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68 -
Enabling tombstone compactions for TWCS
INFO [main] 2025-06-23 10:41:39,242 UTC ColumnFamilyStore.java:484 -
Initializing mykeyspace.MyTWCSTable2
{code}
The SSTable is present on disk and at first glance I can not see a problem.
{code:java}
[ec2-user@ip-1-2-3-4 log]$ sudo ls -la
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/ |
grep nb-9978
-rw-r--r-- 1 user user 5711584 May 28 00:35 nb-997878-big-CompressionInfo.db
-rw-r--r-- 1 user user 7899003536 May 28 00:35 nb-997878-big-Data.db
-rw-r--r-- 1 user user 9 May 28 00:35 nb-997878-big-Digest.crc32
-rw-r--r-- 1 user user 6440 May 28 00:35 nb-997878-big-Filter.db
-rw-r--r-- 1 user user 36811583 May 28 00:35 nb-997878-big-Index.db
-rw-r--r-- 1 user user 17827 May 28 00:35 nb-997878-big-Statistics.db
-rw-r--r-- 1 user user 1197 May 28 00:35 nb-997878-big-Summary.db
-rw-r--r-- 1 user user 92 May 28 00:35 nb-997878-big-TOC.txt
[ec2-user@ip-1-2-3-4 log]$
{code}
Also the Cassandra startup just continues and Cassandra is running fine. I see
no further issues on various metrics or in the logs.
was:
Hi,
we did the update from Cassandra 4.1.8 to Cassandra 4.1.9 on a 30-node cluster
in a rolling fashion, however, on 1 node, 1 single SSTable was showing a
NullPointerException.
{code:java}
INFO [main] 2025-06-23 10:41:38,576 UTC ColumnFamilyStore.java:484 -
Initializing mykeyspace.MyTWCSTable
INFO [SSTableBatchOpen:2] 2025-06-23 10:41:38,583 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1008071-big
(8.141GiB)
...
... more "Opening" lines
...
INFO [SSTableBatchOpen:32] 2025-06-23 10:41:38,586 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-997878-big
(7.357GiB)
ERROR [SSTableBatchOpen:27] 2025-06-23 10:41:38,586 UTC
JVMStabilityInspector.java:68 - Exception in thread
Thread[SSTableBatchOpen:27,5,SSTableBatchOpen]
java.lang.NullPointerException: null
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:510)
at
org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:381)
at
org.apache.cassandra.io.sstable.format.SSTableReader$2.run(SSTableReader.java:551)
at
org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124)
at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
Source)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
Source)
at
io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.base/java.lang.Thread.run(Unknown Source)
INFO [SSTableBatchOpen:13] 2025-06-23 10:41:38,594 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1013913-big
(307.642MiB)
INFO [SSTableBatchOpen:27] 2025-06-23 10:41:38,629 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1001329-big
(7.481GiB)
INFO [SSTableBatchOpen:26] 2025-06-23 10:41:38,679 UTC
SSTableReaderBuilder.java:354 - Opening
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1005032-big
(8.213GiB)
DEBUG [main] 2025-06-23 10:41:39,228 UTC DiskBoundaryManager.java:54 -
Refreshing disk boundary cache for mykeyspace.MyTWCSTable
DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:93 - Got
local ranges [<removed-for-readability>] (ringVersion = 30)
DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:57 - Updating
boundaries from null to
DiskBoundaries{directories=[DataDirectory{location=/var/opt/cassandra}],
positions=[max(9223372036854775807)], ringVersion=30, directoriesVersion=0} for
mykeyspace.MyTWCSTable
DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68 -
Enabling tombstone compactions for TWCS
DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68 -
Enabling tombstone compactions for TWCS
INFO [main] 2025-06-23 10:41:39,242 UTC ColumnFamilyStore.java:484 -
Initializing mykeyspace.MyTWCSTable2
{code}
The SSTable is present on disk and at first glance I can not see a problem.
{code:java}
[ec2-user@ip-1-2-3-4 log]$ sudo ls -la
/var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/ |
grep nb-9978
-rw-r--r-- 1 user user 5711584 May 28 00:35 nb-997878-big-CompressionInfo.db
-rw-r--r-- 1 user user 7899003536 May 28 00:35 nb-997878-big-Data.db
-rw-r--r-- 1 user user 9 May 28 00:35 nb-997878-big-Digest.crc32
-rw-r--r-- 1 user user 6440 May 28 00:35 nb-997878-big-Filter.db
-rw-r--r-- 1 user user 36811583 May 28 00:35 nb-997878-big-Index.db
-rw-r--r-- 1 user user 17827 May 28 00:35 nb-997878-big-Statistics.db
-rw-r--r-- 1 user user 1197 May 28 00:35 nb-997878-big-Summary.db
-rw-r--r-- 1 user user 92 May 28 00:35 nb-997878-big-TOC.txt
[ec2-user@ip-1-2-3-4 log]$
{code}
Also it looks like Cassandra is running fine. I see no further issues on
various metrics or in the logs.
> NullPointerException on opening SSTables
> ----------------------------------------
>
> Key: CASSANDRA-20725
> URL: https://issues.apache.org/jira/browse/CASSANDRA-20725
> Project: Apache Cassandra
> Issue Type: Bug
> Reporter: Christoph Schnepf
> Priority: Normal
>
> Hi,
> we did the update from Cassandra 4.1.8 to Cassandra 4.1.9 on a 30-node
> cluster in a rolling fashion, however, on 1 node, 1 single SSTable was
> showing a NullPointerException.
> {code:java}
> INFO [main] 2025-06-23 10:41:38,576 UTC ColumnFamilyStore.java:484 -
> Initializing mykeyspace.MyTWCSTable
> INFO [SSTableBatchOpen:2] 2025-06-23 10:41:38,583 UTC
> SSTableReaderBuilder.java:354 - Opening
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1008071-big
> (8.141GiB)
> ...
> ... more "Opening" lines
> ...
> INFO [SSTableBatchOpen:32] 2025-06-23 10:41:38,586 UTC
> SSTableReaderBuilder.java:354 - Opening
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-997878-big
> (7.357GiB)
> ERROR [SSTableBatchOpen:27] 2025-06-23 10:41:38,586 UTC
> JVMStabilityInspector.java:68 - Exception in thread
> Thread[SSTableBatchOpen:27,5,SSTableBatchOpen]
> java.lang.NullPointerException: null
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:510)
> at
> org.apache.cassandra.io.sstable.format.SSTableReader.open(SSTableReader.java:381)
> at
> org.apache.cassandra.io.sstable.format.SSTableReader$2.run(SSTableReader.java:551)
> at
> org.apache.cassandra.concurrent.FutureTask$2.call(FutureTask.java:124)
> at org.apache.cassandra.concurrent.FutureTask.call(FutureTask.java:61)
> at org.apache.cassandra.concurrent.FutureTask.run(FutureTask.java:71)
> at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown
> Source)
> at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown
> Source)
> at
> io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> at java.base/java.lang.Thread.run(Unknown Source)
> INFO [SSTableBatchOpen:13] 2025-06-23 10:41:38,594 UTC
> SSTableReaderBuilder.java:354 - Opening
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1013913-big
> (307.642MiB)
> INFO [SSTableBatchOpen:27] 2025-06-23 10:41:38,629 UTC
> SSTableReaderBuilder.java:354 - Opening
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1001329-big
> (7.481GiB)
> INFO [SSTableBatchOpen:26] 2025-06-23 10:41:38,679 UTC
> SSTableReaderBuilder.java:354 - Opening
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/nb-1005032-big
> (8.213GiB)
> DEBUG [main] 2025-06-23 10:41:39,228 UTC DiskBoundaryManager.java:54 -
> Refreshing disk boundary cache for mykeyspace.MyTWCSTable
> DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:93 - Got
> local ranges [<removed-for-readability>] (ringVersion = 30)
> DEBUG [main] 2025-06-23 10:41:39,235 UTC DiskBoundaryManager.java:57 -
> Updating boundaries from null to
> DiskBoundaries{directories=[DataDirectory{location=/var/opt/cassandra}],
> positions=[max(9223372036854775807)], ringVersion=30, directoriesVersion=0}
> for mykeyspace.MyTWCSTable
> DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68
> - Enabling tombstone compactions for TWCS
> DEBUG [main] 2025-06-23 10:41:39,236 UTC TimeWindowCompactionStrategy.java:68
> - Enabling tombstone compactions for TWCS
> INFO [main] 2025-06-23 10:41:39,242 UTC ColumnFamilyStore.java:484 -
> Initializing mykeyspace.MyTWCSTable2
> {code}
> The SSTable is present on disk and at first glance I can not see a problem.
> {code:java}
> [ec2-user@ip-1-2-3-4 log]$ sudo ls -la
> /var/opt/cassandra/mykeyspace/MyTWCSTable-ed2fa920c27411eabe8a17a890aa6eb4/ |
> grep nb-9978
> -rw-r--r-- 1 user user 5711584 May 28 00:35
> nb-997878-big-CompressionInfo.db
> -rw-r--r-- 1 user user 7899003536 May 28 00:35 nb-997878-big-Data.db
> -rw-r--r-- 1 user user 9 May 28 00:35 nb-997878-big-Digest.crc32
> -rw-r--r-- 1 user user 6440 May 28 00:35 nb-997878-big-Filter.db
> -rw-r--r-- 1 user user 36811583 May 28 00:35 nb-997878-big-Index.db
> -rw-r--r-- 1 user user 17827 May 28 00:35 nb-997878-big-Statistics.db
> -rw-r--r-- 1 user user 1197 May 28 00:35 nb-997878-big-Summary.db
> -rw-r--r-- 1 user user 92 May 28 00:35 nb-997878-big-TOC.txt
> [ec2-user@ip-1-2-3-4 log]$
> {code}
> Also the Cassandra startup just continues and Cassandra is running fine. I
> see no further issues on various metrics or in the logs.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
