[ https://issues.apache.org/jira/browse/IGNITE-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladislav Pyatkov updated IGNITE-22598: --------------------------------------- Description: h3. Motivation Many exception might appear in log of the thrutput test. After the partition storage is in undefined state. Notsurprised the te continue work with the storage leads to another issues. {noformat} 2024-06-27 12:19:46:881 +0300 [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] Error occurred on a user's state machine org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123) at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336) at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317) at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109) at org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83) at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270) at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114) at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165) at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668) at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155) at org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166) at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458) at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131) at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125) at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326) at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on search row: org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709] at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660) at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109) at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119) ... 28 more Caused by: org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate temporary buffer for checkpoint (increase checkpointPageBufferSize configuration property): default at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434) at org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398) at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347) at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319) at org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193) at org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102) at org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083) ... 29 more {noformat} h3. Definition of done Exceptions like that is in the attached log have not to appear. was: h3. Motivation Many exception might appear in log of the thrutput test. After the partition storage is in undefined state. Notsurprised the te continue work with the storage leads to another issues. {noformat} 2024-06-27 12:19:46:881 +0300 [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] Error occurred on a user's state machine org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123) at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336) at org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317) at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109) at org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83) at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270) at org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114) at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165) at org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668) at org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155) at org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76) at org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) at org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166) at org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539) at org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458) at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131) at org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125) at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326) at org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283) at com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) at com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) at java.base/java.lang.Thread.run(Thread.java:829) Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on search row: org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709] at org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660) at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109) at org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119) ... 28 more Caused by: org.apache.ignite.internal.lang.IgniteInternalException: IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate temporary buffer for checkpoint (increase checkpointPageBufferSize configuration property): default at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449) at org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434) at org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398) at org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347) at org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319) at org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193) at org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102) at org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) at org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) at org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083) ... 29 more {noformat} Definition of done Exceptions like that is in the attached log have not to appear. > Failed to allocate temporary buffer for checkpoint > -------------------------------------------------- > > Key: IGNITE-22598 > URL: https://issues.apache.org/jira/browse/IGNITE-22598 > Project: Ignite > Issue Type: Bug > Reporter: Vladislav Pyatkov > Priority: Major > Labels: ignite-3 > Attachments: > poc-tester-SERVER-192.168.1.41-id-0-2024-06-27-09-14-17-client.log.2 > > > h3. Motivation > Many exception might appear in log of the thrutput test. After the partition > storage is in undefined state. Notsurprised the te continue work with the > storage leads to another issues. > {noformat} > 2024-06-27 12:19:46:881 +0300 > [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor] > Error occurred on a user's state machine > org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1 > TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index > at > org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123) > at > org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336) > at > org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317) > at > org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109) > at > org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83) > at > org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270) > at > org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69) > at > org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173) > at > org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114) > at > org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165) > at > org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668) > at > org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155) > at > org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76) > at > org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109) > at > org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289) > at > org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209) > at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133) > at > org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166) > at > org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702) > at > org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571) > at > org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539) > at > org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458) > at > org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131) > at > org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125) > at > org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326) > at > org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283) > at > com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167) > at > com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122) > at java.base/java.lang.Thread.run(Thread.java:829) > Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException: > IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is > corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime > failure on search row: > org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709] > at > org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109) > at > org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119) > ... 28 more > Caused by: org.apache.ignite.internal.lang.IgniteInternalException: > IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate > temporary buffer for checkpoint (increase checkpointPageBufferSize > configuration property): default > at > org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099) > at > org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084) > at > org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449) > at > org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434) > at > org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398) > at > org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347) > at > org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193) > at > org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102) > at > org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161) > at > org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083) > ... 29 more > {noformat} > h3. Definition of done > Exceptions like that is in the attached log have not to appear. -- This message was sent by Atlassian Jira (v8.20.10#820010)