[
https://issues.apache.org/jira/browse/IGNITE-22598?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Vladislav Pyatkov updated IGNITE-22598:
---------------------------------------
Description:
h3. Motivation
Many exception might appear in log of the thrutput test. After the partition
storage is in undefined state. Notsurprised the te continue work with the
storage leads to another issues.
{noformat}
2024-06-27 12:19:46:881 +0300
[INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor]
Error occurred on a user's state machine
org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
at
org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
at
org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
at
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted
[groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on
search row:
org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
... 28 more
Caused by: org.apache.ignite.internal.lang.IgniteInternalException:
IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate
temporary buffer for checkpoint (increase checkpointPageBufferSize
configuration property): default
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
at
org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
at
org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
at
org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
... 29 more
{noformat}
h3. Definition of done
Exceptions like that is in the attached log have not to appear.
was:
h3. Motivation
Many exception might appear in log of the thrutput test. After the partition
storage is in undefined state. Notsurprised the te continue work with the
storage leads to another issues.
{noformat}
2024-06-27 12:19:46:881 +0300
[INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor]
Error occurred on a user's state machine
org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
at
org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
at
org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
at
org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
at
org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
at
org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
at
org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
at
org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
at
org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
at
org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
at
org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
at
org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
at
com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
at
com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
at java.base/java.lang.Thread.run(Thread.java:829)
Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is corrupted
[groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime failure on
search row:
org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
at
org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
at
org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
... 28 more
Caused by: org.apache.ignite.internal.lang.IgniteInternalException:
IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate
temporary buffer for checkpoint (increase checkpointPageBufferSize
configuration property): default
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
at
org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
at
org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
at
org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
at
org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
at
org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
at
org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
... 29 more
{noformat}
Definition of done
Exceptions like that is in the attached log have not to appear.
> Failed to allocate temporary buffer for checkpoint
> --------------------------------------------------
>
> Key: IGNITE-22598
> URL: https://issues.apache.org/jira/browse/IGNITE-22598
> Project: Ignite
> Issue Type: Bug
> Reporter: Vladislav Pyatkov
> Priority: Major
> Labels: ignite-3
> Attachments:
> poc-tester-SERVER-192.168.1.41-id-0-2024-06-27-09-14-17-client.log.2
>
>
> h3. Motivation
> Many exception might appear in log of the thrutput test. After the partition
> storage is in undefined state. Notsurprised the te continue work with the
> storage leads to another issues.
> {noformat}
> 2024-06-27 12:19:46:881 +0300
> [INFO][%poc-tester-SERVER-192.168.1.41-id-0%JRaft-FSMCaller-Disruptor_stripe_6-0][ActionRequestProcessor]
> Error occurred on a user's state machine
> org.apache.ignite.internal.storage.StorageException: IGN-STORAGE-1
> TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to put value into index
> at
> org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:123)
> at
> org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busy(AbstractPageMemoryIndexStorage.java:336)
> at
> org.apache.ignite.internal.storage.pagememory.index.AbstractPageMemoryIndexStorage.busyNonDataRead(AbstractPageMemoryIndexStorage.java:317)
> at
> org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.put(PageMemoryHashIndexStorage.java:109)
> at
> org.apache.ignite.internal.table.distributed.TableSchemaAwareIndexStorage.put(TableSchemaAwareIndexStorage.java:83)
> at
> org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.putToIndex(IndexUpdateHandler.java:270)
> at
> org.apache.ignite.internal.table.distributed.index.IndexUpdateHandler.addToIndexes(IndexUpdateHandler.java:69)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.tryProcessRow(StorageUpdateHandler.java:173)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.lambda$handleUpdate$0(StorageUpdateHandler.java:114)
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.lambda$runConsistently$0(PersistentPageMemoryMvPartitionStorage.java:165)
> at
> org.apache.ignite.internal.storage.pagememory.mv.AbstractPageMemoryMvPartitionStorage.busy(AbstractPageMemoryMvPartitionStorage.java:668)
> at
> org.apache.ignite.internal.storage.pagememory.mv.PersistentPageMemoryMvPartitionStorage.runConsistently(PersistentPageMemoryMvPartitionStorage.java:155)
> at
> org.apache.ignite.internal.table.distributed.raft.snapshot.outgoing.SnapshotAwarePartitionDataStorage.runConsistently(SnapshotAwarePartitionDataStorage.java:76)
> at
> org.apache.ignite.internal.table.distributed.StorageUpdateHandler.handleUpdate(StorageUpdateHandler.java:109)
> at
> org.apache.ignite.internal.table.distributed.raft.PartitionListener.handleUpdateCommand(PartitionListener.java:289)
> at
> org.apache.ignite.internal.table.distributed.raft.PartitionListener.lambda$onWrite$1(PartitionListener.java:209)
> at java.base/java.util.Iterator.forEachRemaining(Iterator.java:133)
> at
> org.apache.ignite.internal.table.distributed.raft.PartitionListener.onWrite(PartitionListener.java:166)
> at
> org.apache.ignite.internal.raft.server.impl.JraftServerImpl$DelegatingStateMachine.onApply(JraftServerImpl.java:702)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doApplyTasks(FSMCallerImpl.java:571)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.doCommitted(FSMCallerImpl.java:539)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl.runApplyTask(FSMCallerImpl.java:458)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:131)
> at
> org.apache.ignite.raft.jraft.core.FSMCallerImpl$ApplyTaskHandler.onEvent(FSMCallerImpl.java:125)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:326)
> at
> org.apache.ignite.raft.jraft.disruptor.StripedDisruptor$StripeEntryHandler.onEvent(StripedDisruptor.java:283)
> at
> com.lmax.disruptor.BatchEventProcessor.processEvents(BatchEventProcessor.java:167)
> at
> com.lmax.disruptor.BatchEventProcessor.run(BatchEventProcessor.java:122)
> at java.base/java.lang.Thread.run(Thread.java:829)
> Caused by: org.apache.ignite.internal.pagememory.tree.CorruptedTreeException:
> IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d B+Tree is
> corrupted [groupId=10, pageIds=[562988608185341], groupName=10, msg=Runtime
> failure on search row:
> org.apache.ignite.internal.storage.pagememory.index.hash.HashIndexRow@60f9a709]
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.corruptedTreeException(BplusTree.java:6660)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2109)
> at
> org.apache.ignite.internal.storage.pagememory.index.hash.PageMemoryHashIndexStorage.lambda$put$1(PageMemoryHashIndexStorage.java:119)
> ... 28 more
> Caused by: org.apache.ignite.internal.lang.IgniteInternalException:
> IGN-CMN-65535 TraceId:0d512917-7a88-4a7c-94c9-03d86304997d Failed to allocate
> temporary buffer for checkpoint (increase checkpointPageBufferSize
> configuration property): default
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.postWriteLockPage(PersistentPageMemory.java:1099)
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLockPage(PersistentPageMemory.java:1084)
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:449)
> at
> org.apache.ignite.internal.pagememory.persistence.PersistentPageMemory.writeLock(PersistentPageMemory.java:434)
> at
> org.apache.ignite.internal.pagememory.util.PageHandler.writeLock(PageHandler.java:398)
> at
> org.apache.ignite.internal.pagememory.util.PageHandler.writePage(PageHandler.java:347)
> at
> org.apache.ignite.internal.pagememory.datastructure.DataStructure.write(DataStructure.java:319)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.access$5000(BplusTree.java:193)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$Put.tryInsert(BplusTree.java:4102)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree$Invoke.onNotFound(BplusTree.java:4360)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2186)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invokeDown(BplusTree.java:2161)
> at
> org.apache.ignite.internal.pagememory.tree.BplusTree.invoke(BplusTree.java:2083)
> ... 29 more
> {noformat}
> h3. Definition of done
> Exceptions like that is in the attached log have not to appear.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
