[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-17933:
---
Fix Version/s: (was: 4.1-rc)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.7, 4.1-rc1, 4.1, 4.2
>
> Attachments: signature.asc
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.ca
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-17933:
---
Fix Version/s: 4.1-rc1
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.7, 4.1-rc1, 4.1-rc, 4.1, 4.2
>
> Attachments: signature.asc
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cass
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-17933:
---
Fix Version/s: 4.1
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.7, 4.1-rc, 4.1, 4.2
>
> Attachments: signature.asc
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.B
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Fix Version/s: 4.0.7
4.1-rc
4.2
(was: 4.x)
(was: 4.0.x)
(was: 4.1.x)
Source Control Link:
https://github.com/apache/cassandra/commit/488c0c75a8f632f2db4e3db39f2ebcf8a489971e
Resolution: Fixed
Status: Resolved (was: Ready to Commit)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.7, 4.1-rc, 4.2
>
> Attachments: signature.asc
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChron
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
miklosovic updated CASSANDRA-17933:
---
Attachment: signature.asc
that is probably some leftover from times I was rebasing stuff, no worries.
Same thing just called differently.
Sent from ProtonMail mobile
\
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
> Attachments: signature.asc
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.B
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: Review In Progress (was: Needs Committer)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common frame
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: Ready to Commit (was: Review In Progress)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common frame
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: Needs Committer (was: Patch Available)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common frames o
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: Patch Available (was: In Progress)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common frames omitt
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: Changes Suggested (was: Review In Progress)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common fra
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Status: In Progress (was: Changes Suggested)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
> ... 11 common frames omi
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Caleb Rackliffe updated CASSANDRA-17933:
Reviewers: Caleb Rackliffe, Caleb Rackliffe
Caleb Rackliffe, Caleb Rackliffe (was: Caleb Rackliffe)
Status: Review In Progress (was: Patch Available)
Sure, I'll read through the comments and take a look at the patch.
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Stefan Miklosovic updated CASSANDRA-17933:
--
Test and Documentation Plan: junits + dtest
Status: Patch Available (was: In Progress)
This is what I have for 4.0 (will be applied to 4.1 and trunk too)
cassandra-4.0 patch [https://github.com/apache/cassandra/pull/1894]
cassandra-4.0 build
[https://ci-cassandra.apache.org/view/patches/job/Cassandra-devbranch/1964/]
The patch also backports what is in CASSANDRA-17595. I am not sure what it was
fixed for 4.1 and trunk only, the issue in 17959 is present in 4.0 too. I
noticed this while I was patching this issue.
The approach I took cleans empty cq4 files on every BinLog start. _cleaning of
log dir_ is different from {_}cleaning empty cq4 files{_}. The former happens
only explicitly on request, the latter happens every time when BinLog is about
to be started because it just fails with empty cq4 files present.
I have also consolidated the cleanup methods and extracted common parts for
reusability.
The related dtest is here https://github.com/apache/cassandra-dtest/pull/203
On general consensus what I am trying to do here I will prepare two remaining
branches with builds.
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFi
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brandon Williams updated CASSANDRA-17933:
-
Component/s: Local/Startup and Shutdown
Status: Open (was: Triage Needed)
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
> Components: Local/Startup and Shutdown
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
> at
> org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Michael Semb Wever updated CASSANDRA-17933:
---
Bug Category: Parent values: Availability(12983)Level 1 values: Process
Crash(12992)
Complexity: Normal
Discovered By: User Report
Fix Version/s: 4.0.x
4.1.x
4.x
Severity: Normal
Since Version: 4.0.0
> Zero length file in Audit log folder, prevents a node from starting
> ---
>
> Key: CASSANDRA-17933
> URL: https://issues.apache.org/jira/browse/CASSANDRA-17933
> Project: Cassandra
> Issue Type: Bug
>Reporter: Andrew Hogg
>Assignee: Stefan Miklosovic
>Priority: Normal
> Fix For: 4.0.x, 4.1.x, 4.x
>
>
> We have encountered a 4.0.3 cluster where the audit log folder had a zero
> byte length file within it after the node had stopped. It is not clear how
> Cassandra got to the point of this file existing. On restarting the node, the
> node will not start and throws the following stack trace.
> {code:java}
> ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
> encountered during startup
> java.lang.ExceptionInInitializerError: null
> at
> org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
> at
> org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
> at
> org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
> Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
> create instance of IAuditLogger.
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
> at
> org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
> at
> org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
> ... 3 common frames omitted
> Caused by: java.lang.reflect.InvocationTargetException: null
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
> ... 6 common frames omitted
> Caused by: java.nio.channels.OverlappingFileLockException: null
> at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
> at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
> at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
> at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
> at
> net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
> at
> net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
> at
> net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
> at
> net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
> at
> net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
> at
> net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
> at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
> at java.lang.ThreadLocal.get(ThreadLocal.java:170)
> at
> net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
> at
> net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
> at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
> at
> org.apache.cass
[jira] [Updated] (CASSANDRA-17933) Zero length file in Audit log folder, prevents a node from starting
[
https://issues.apache.org/jira/browse/CASSANDRA-17933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Hogg updated CASSANDRA-17933:
Description:
We have encountered a 4.0.3 cluster where the audit log folder had a zero byte
length file within it after the node had stopped. It is not clear how Cassandra
got to the point of this file existing. On restarting the node, the node will
not start and throws the following stack trace.
{code:java}
ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
encountered during startup
java.lang.ExceptionInInitializerError: null
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:468)
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:765)
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:889)
Caused by: org.apache.cassandra.exceptions.ConfigurationException: Unable to
create instance of IAuditLogger.
at
org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:686)
at
org.apache.cassandra.audit.AuditLogManager.getAuditLogger(AuditLogManager.java:95)
at
org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:74)
at
org.apache.cassandra.audit.AuditLogManager.(AuditLogManager.java:60)
... 3 common frames omitted
Caused by: java.lang.reflect.InvocationTargetException: null
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
at
org.apache.cassandra.utils.FBUtilities.newAuditLogger(FBUtilities.java:682)
... 6 common frames omitted
Caused by: java.nio.channels.OverlappingFileLockException: null
at sun.nio.ch.SharedFileLockTable.checkList(FileLockTable.java:255)
at sun.nio.ch.SharedFileLockTable.add(FileLockTable.java:152)
at sun.nio.ch.FileChannelImpl.lock(FileChannelImpl.java:1068)
at java.nio.channels.FileChannel.lock(FileChannel.java:1053)
at
net.openhft.chronicle.bytes.MappedFile.resizeRafIfTooSmall(MappedFile.java:369)
at
net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:307)
at
net.openhft.chronicle.bytes.MappedFile.acquireByteStore(MappedFile.java:269)
at
net.openhft.chronicle.bytes.MappedBytes.acquireNextByteStore0(MappedBytes.java:434)
at
net.openhft.chronicle.bytes.MappedBytes.readVolatileInt(MappedBytes.java:792)
at
net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.headerRecovery(SingleChronicleQueue.java:1027)
at
net.openhft.chronicle.queue.impl.single.SingleChronicleQueue$StoreSupplier.acquire(SingleChronicleQueue.java:981)
at
net.openhft.chronicle.queue.impl.WireStorePool.acquire(WireStorePool.java:53)
at
net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.cleanupStoreFilesWithNoData(SingleChronicleQueue.java:821)
at
net.openhft.chronicle.queue.impl.single.StoreAppender.(StoreAppender.java:75)
at
net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.newAppender(SingleChronicleQueue.java:422)
at
net.openhft.chronicle.core.threads.CleaningThreadLocal.initialValue(CleaningThreadLocal.java:54)
at java.lang.ThreadLocal.setInitialValue(ThreadLocal.java:180)
at java.lang.ThreadLocal.get(ThreadLocal.java:170)
at
net.openhft.chronicle.core.threads.CleaningThreadLocal.get(CleaningThreadLocal.java:59)
at
net.openhft.chronicle.queue.impl.single.SingleChronicleQueue.acquireAppender(SingleChronicleQueue.java:441)
at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:133)
at org.apache.cassandra.utils.binlog.BinLog.(BinLog.java:65)
at
org.apache.cassandra.utils.binlog.BinLog$Builder.build(BinLog.java:453)
at
org.apache.cassandra.audit.BinAuditLogger.(BinAuditLogger.java:55)
... 11 common frames omitted {code}
To reproduce, we place a zero length file and attempted to start the node, and
saw the same stack trace.
{code:java}
ll ../logs/audit/
total 4
rw-rw-r-. 1 automaton automaton 0 Sep 28 13:00 20220928-12.cq4
rw-rw-r-. 1 automaton automaton 131072 Sep 28 13:00 metadata.cq4t {code}
was:
We have encountered a 4.0.3 cluster where the audit log folder had a zero byte
length file within it after the node had stopped. It is not clear how Cassandra
got to the point of this file existing. On restarting the node, the node will
not start and throws the following stack trace.
{code:java}
ERROR [main] 2022-09-26 14:01:27,892 CassandraDaemon.java:911 - Exception
encountered during startup
java.lang.Exception
