Hi Team, We are trying to start another Ignite cluster by taking a copy of the running cluster's datastore (source cluster's datastore is getting modified in parallel). So, when we try to start the server node with copied datastore, it gives error as below. Also, giving cluster configuration for reference:
pageSize=#{4 * 1024} walMode=LOG_ONLY walFlushFrequency=60000 rebalanceThreadPoolSize=8 rebalanceThrottle=100 rebalanceBatchSize=#{32 * 1024 * 1024} storagePath=/datastore/datastore walPath=/datastore1/wal walArchivePath=/datastore1/archive metadataWorkDir=/datastore/metadataWorkDir [2019-06-05T12:21:52,943][INFO ][main][GridCacheDatabaseSharedManager] Read checkpoint status [startMarker=null, endMarker=null] [2019-06-05T12:21:52,967][INFO ][main][PageMemoryImpl] Started page memory [memoryAllocated=128.0 MiB, pages=31744, tableSize=2.5 MiB, checkpointBuffer=100.0 MiB] [2019-06-05T12:21:52,968][INFO ][main][GridCacheDatabaseSharedManager] Checking memory state [lastValidPos=FileWALPointer [idx=0, fileOff=0, len=0], lastMarked=FileWALPointer [idx=0, fileOff=0, len=0], lastCheckpointId=00000000-0000-0000-0000-000000000000] [2019-06-05T12:21:52,973][ERROR][main][IgniteKernal%EDIFCustomer_DR] Exception during start processors, node will be stopped and close connections org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1742) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:980) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.Ignition.start(Ignition.java:352) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) [ignite-core-2.6.0.jar:2.6.0] Caused by: org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@4d6c2], start=FileWALPointer [idx=0, fileOff=0, len=0]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:3009) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2960) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2896) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:799) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1968) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:574) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:525) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:700) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739) ~[ignite-core-2.6.0.jar:2.6.0] ... 11 more [2019-06-05T12:21:52,978][ERROR][main][IgniteKernal%EDIFCustomer_DR] Got exception while starting (will rollback startup routine). org.apache.ignite.IgniteCheckedException: Failed to start processor: GridProcessorAdapter [] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1742) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.start(IgniteKernal.java:980) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start0(IgnitionEx.java:2014) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx$IgniteNamedInstance.start(IgnitionEx.java:1723) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start0(IgnitionEx.java:1151) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.startConfigurations(IgnitionEx.java:1069) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:955) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:854) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:724) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgnitionEx.start(IgnitionEx.java:693) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.Ignition.start(Ignition.java:352) [ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.startup.cmdline.CommandLineStartup.main(CommandLineStartup.java:301) [ignite-core-2.6.0.jar:2.6.0] Caused by: org.apache.ignite.IgniteCheckedException: WAL history is too short [descs=[org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$FileDescriptor@4d6c2], start=FileWALPointer [idx=0, fileOff=0, len=0]] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.init(FileWriteAheadLogManager.java:3009) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2960) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager$RecordsIterator.<init>(FileWriteAheadLogManager.java:2896) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.wal.FileWriteAheadLogManager.replay(FileWriteAheadLogManager.java:799) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.restoreMemory(GridCacheDatabaseSharedManager.java:1968) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.readMetastore(GridCacheDatabaseSharedManager.java:574) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.persistence.GridCacheDatabaseSharedManager.start0(GridCacheDatabaseSharedManager.java:525) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheSharedManagerAdapter.start(GridCacheSharedManagerAdapter.java:61) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.processors.cache.GridCacheProcessor.start(GridCacheProcessor.java:700) ~[ignite-core-2.6.0.jar:2.6.0] at org.apache.ignite.internal.IgniteKernal.startProcessor(IgniteKernal.java:1739) ~[ignite-core-2.6.0.jar:2.6.0] So, is there any way to start this cluster with copied data store forcefully? This scenario may also arrive if WAL disk gets failed. How can we atleast start the cluster with minimum data loss ? Any help would be highly appreciated ! Thanks and Regards, Kamlesh Joshi "Confidentiality Warning: This message and any attachments are intended only for the use of the intended recipient(s). are confidential and may be privileged. If you are not the intended recipient. you are hereby notified that any review. re-transmission. conversion to hard copy. copying. circulation or other use of this message and any attachments is strictly prohibited. If you are not the intended recipient. please notify the sender immediately by return email. and delete this message and any attachments from your system. Virus Warning: Although the company has taken reasonable precautions to ensure no viruses are present in this email. The company cannot accept responsibility for any loss or damage arising from the use of this email or attachment."