[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830431#comment-17830431 ] Berenguer Blasi commented on CASSANDRA-19471: - If so +1 > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17830395#comment-17830395 ] Berenguer Blasi commented on CASSANDRA-19471: - Review is for these correct? - PR https://github.com/driftx/cassandra/commit/3659730ac1cfcadada46677b19b5e714531a2984 - dtests PR https://github.com/driftx/cassandra-dtest/commit/2499a609b7a8bd750417c3fa56855ab1e9e800f1 - CI https://app.circleci.com/pipelines/github/driftx/cassandra/1536/workflows/150c6753-e905-4503-9f58-f6c7e8357881 > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829919#comment-17829919 ] Brandon Williams commented on CASSANDRA-19471: -- I believe the root cause is that direct mode initially allocates only the block size on disk (4k in this case) while legacy allocates the full segment size (1MB). This leaves legacy mode with enough headroom for mutations like schema changes without changing the on-disk size, while direct mode will reflect them. This is good news though, because it shows the test is faulty. First, it shouldn't begin measuring the CL until _after_ the schema is created since those mutations are always durable. Second, it needs to reset the batch mode options when it recreates the cluster for the second run, otherwise the CL gets an asynchronous flush that breaks the test again. I've made those changes [here|https://github.com/driftx/cassandra-dtest/commit/2499a609b7a8bd750417c3fa56855ab1e9e800f1] and [here|https://app.circleci.com/pipelines/github/driftx/cassandra/1536/workflows/150c6753-e905-4503-9f58-f6c7e8357881] is a circle run with repeats for latest/non-latest to make sure the test still works, and a full latest gamut to make sure nothing else was broken by the direct CL changes. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829805#comment-17829805 ] Branimir Lambov commented on CASSANDRA-19471: - They are only for the IAE, which is a a serious issue and IMHO a blocker for 5.0. I have not investigated the commitlog being written with durable writes off which is a much more benign issue. It is likely caused by the preparation of the direct I/O segments writing and flushing the header and first sync marker in advance of any use of the segment. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829641#comment-17829641 ] Brandon Williams commented on CASSANDRA-19471: -- Thank you for further suggestions, [~blambov], I have implemented them [here|https://github.com/driftx/cassandra/commit/3659730ac1cfcadada46677b19b5e714531a2984] but it's not entirely clear to me if this is supposed to only fix the IAE (it does) or that and also the commitlog being written when durable writes are disabled (it does not.) > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Assignee: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829282#comment-17829282 ] Brandon Williams commented on CASSANDRA-19471: -- bq. I doubt this is a home run I guess I wasn't wrong, this [fails horribly|https://app.circleci.com/pipelines/github/driftx/cassandra/1533/workflows/b81cb765-6f9f-44c0-ae36-83158bc208d2/jobs/78867] in circle, though oddly I can't reproduce that anywhere else. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829278#comment-17829278 ] Brandon Williams commented on CASSANDRA-19471: -- I did that, but the problem is there are no repeat jobs generated. It looks like it's not just latest but all of them. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829250#comment-17829250 ] Ekaterina Dimitrova commented on CASSANDRA-19471: - {quote}it looks like the 'latest repeat' is now gone and I don't see a way to trigger it in generate.sh {quote} It replaces the offheap which never had a dedicated environment variable either. We use the REPEATED_DTESTS one for vnode, no_vnode, latest(former offheap), etc. {code:java} .circleci/generate.sh -ps -e REPEATED_DTESTS=configuration_test.py::TestConfiguration::test_change_durable_writes{code} > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829242#comment-17829242 ] Brandon Williams commented on CASSANDRA-19471: -- I doubt this is a home run since I am not very familiar with this code at all, but the stab I took [here|https://github.com/driftx/cassandra/commits/CASSANDRA-19471-5.0/] does fix the test. Which I would prove on circle except it looks like the 'latest repeat' is now gone and I don't see a way to trigger it in generate.sh. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829239#comment-17829239 ] Stefan Miklosovic commented on CASSANDRA-19471: --- I am not planning to be involved. Feel free to take it. Thank you very much. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17829201#comment-17829201 ] Ekaterina Dimitrova commented on CASSANDRA-19471: - [~blambov] , thank you for looking into this and for providing pointers. [~smiklosovic] , [~Bereng] , [~brandon.williams] , does anyone of you plan to look into it? I can take it tomorrow if no one assigns it, but I wanted to confirm no one is on top of it yet, as I see several people commenting, etc. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17828364#comment-17828364 ] Branimir Lambov commented on CASSANDRA-19471: - I believe the problem is that the buffer's limit (set [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/DirectIOSegment.java#L208]) is not the same as the buffer's capacity (from which `endOfBuffer` is set [here|https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/commitlog/CommitLogSegment.java#L177]). I guess what we want is to change the former to set the limit first and then apply `slice`. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827464#comment-17827464 ] Brandon Williams commented on CASSANDRA-19471: -- The IAE seems to be caused by the segment size being set to 1MB, at 2MB it doesn't hasn't occurred but we still write the commitlog when we shouldn't and fail. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827446#comment-17827446 ] Stefan Miklosovic commented on CASSANDRA-19471: --- [~blambov] I tried to make the sense of it here (1). It is very suspicious that we are so far in the control flow, on face value one would say that if it is not possible to allocate required space in the current segment, it would return null and it would keep trying until it does. It just should not go so far at all ... (1) https://issues.apache.org/jira/browse/CASSANDRA-19465?focusedCommentId=17827038&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17827038 > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827392#comment-17827392 ] Berenguer Blasi commented on CASSANDRA-19471: - Ah that feeling when a though test is fixed on one email... and next email is just that test being reopened :-DDD > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827281#comment-17827281 ] Stefan Miklosovic commented on CASSANDRA-19471: --- ah great! I have not noticed this was happening on -latest in 19465 ... Anyway, at least it discovered other issues and we improved both the test and pinned down testing versions :) cc [~Bereng] > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827278#comment-17827278 ] Brandon Williams commented on CASSANDRA-19471: -- After much fretting on CASSANDRA-19465, I now realize that the original report was actually against a -latest config circle run, so this all makes sense. > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, and the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org
[jira] [Commented] (CASSANDRA-19471) Commitlog with direct io fails test_change_durable_writes
[ https://issues.apache.org/jira/browse/CASSANDRA-19471?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17827277#comment-17827277 ] Brandon Williams commented on CASSANDRA-19471: -- /cc [~blambov] > Commitlog with direct io fails test_change_durable_writes > - > > Key: CASSANDRA-19471 > URL: https://issues.apache.org/jira/browse/CASSANDRA-19471 > Project: Cassandra > Issue Type: Bug > Components: Local/Commit Log >Reporter: Brandon Williams >Priority: Normal > Fix For: 5.0-rc, 5.x > > > With the commitlog_disk_access_mode set to direct, the improved > configuration_test.py::TestConfiguration::test_change_durable_writes from > CASSANDRA-19465, this fails with either: > {noformat} > AssertionError: Commitlog was written with durable writes disabled > {noformat} > Or what appears to be the original exception reported in CASSANDRA-19465: > {noformat} > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,465 > StorageService.java:631 - Stopping native transport > node1: ERROR [MutationStage-5] 2024-03-14 17:16:08,465 > StorageProxy.java:1670 - Failed to apply mutation locally : > java.lang.IllegalArgumentException: newPosition > limit: (1048634 > 1048576) > at java.base/java.nio.Buffer.createPositionException(Buffer.java:341) > at java.base/java.nio.Buffer.position(Buffer.java:316) > at java.base/java.nio.ByteBuffer.position(ByteBuffer.java:1516) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:321) > at > java.base/java.nio.MappedByteBuffer.position(MappedByteBuffer.java:73) > at > org.apache.cassandra.db.commitlog.CommitLogSegment.allocate(CommitLogSegment.java:216) > at > org.apache.cassandra.db.commitlog.CommitLogSegmentManagerStandard.allocate(CommitLogSegmentManagerStandard.java:52) > at org.apache.cassandra.db.commitlog.CommitLog.add(CommitLog.java:307) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.addToCommitLog(CassandraKeyspaceWriteHandler.java:99) > at > org.apache.cassandra.db.CassandraKeyspaceWriteHandler.beginWrite(CassandraKeyspaceWriteHandler.java:53) > at org.apache.cassandra.db.Keyspace.applyInternal(Keyspace.java:612) > at org.apache.cassandra.db.Keyspace.apply(Keyspace.java:497) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:244) > at org.apache.cassandra.db.Mutation.apply(Mutation.java:264) > at > org.apache.cassandra.service.StorageProxy$4.runMayThrow(StorageProxy.java:1664) > at > org.apache.cassandra.service.StorageProxy$LocalMutationRunnable.run(StorageProxy.java:2624) > at > org.apache.cassandra.concurrent.ExecutionFailure$2.run(ExecutionFailure.java:163) > at org.apache.cassandra.concurrent.SEPWorker.run(SEPWorker.java:143) > at > io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30) > at java.base/java.lang.Thread.run(Thread.java:833) > node1: ERROR [PERIODIC-COMMIT-LOG-SYNCER] 2024-03-14 17:16:08,470 > StorageService.java:636 - Stopping gossiper > {noformat} -- This message was sent by Atlassian Jira (v8.20.10#820010) - To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org For additional commands, e-mail: commits-h...@cassandra.apache.org