[ https://issues.apache.org/jira/browse/IGNITE-21656?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vladimir Steshin updated IGNITE-21656: -------------------------------------- Description: Ignite may fail to create dump of cache with a node filter. 1) If there are no partitions on some node, we assign the empty set as the snapshot task result: {code:java} if (parts.isEmpty() && !withMetaStorage) task0 = new GridFinishedFuture<>(Collections.emptySet()); else { ... } {code} Cache dump doesn't contain the metastorage. Further, we may cast the result to {code:java} SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result(); {code} but it is a {code:java} Collections#EMPTY_SET {code} Stacktrace: {code:java} [ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be cast to class o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap'; o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in unnamed module of loader 'app')]] java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be cast to class org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap'; org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in unnamed module of loader 'app') at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221) ~[classes/:?] at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) ~[classes/:?] at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) ~[classes/:?] {code} 2) Even we fix the cast, further failure appears: {code:java} private GridKernalContext standaloneKernalContext(File dumpDir, IgniteLogger log) { File binaryMeta = CacheObjectBinaryProcessorImpl.binaryWorkDir(dumpDir.getAbsolutePath(), F.first(metadata).folderName()); File marshaller = new File(dumpDir, DFLT_MARSHALLER_PATH); A.ensure(binaryMeta.exists(), "binary metadata directory not exists"); {code} If cache with node filter was created by a client or not by the excluded node (or may be not from the coordinator), it may be checked before any dump data are written. Stacktrace: {code:java} Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is invalid: binary metadata directory not exists at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254) at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) ... 3 more Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid: binary metadata directory not exists at org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.(Dump.java:149) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370) {code} 3) We might fix this with creating normal dump task instead of {code:java} task0 = new GridFinishedFuture<>(new SnapshotFutureTaskResult(Collections.emptySet(), null)) {code} It could write the nessesary dump content before checking it. But one more fauilure is met: {code:java} Caused by: class org.apache.ignite.IgniteException: Wrong number of group directories: 0 at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpGroupDirectory(Dump.java:384) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.configs(Dump.java:237) at org.apache.ignite.dump.DumpReader.lambda$run$2(DumpReader.java:107) at java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) at java.base/java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1785) at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294) at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206) at java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161) at java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300) at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.AbstractCacheDumpTest$1.onCacheConfigs(AbstractCacheDumpTest.java:370) at org.apache.ignite.dump.DumpReader.run(DumpReader.java:106) ... 12 more {code} There is no cache group directory at all on the filtered node. was: Ignite fails to create dump of cache with a node filter. If there is no partitions on some cone, we assign the empty sat an the snapshot task result: {code:java} if (parts.isEmpty() && !withMetaStorage) task0 = new GridFinishedFuture<>(Collections.emptySet()); else { ... } {code} Cache dump doesn't contain the metastoreage. Further, we may cast the result to {code:java} SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result(); {code} but it is a {code:java} Collections#EMPTY_SET {code} Stacktrace: {code:java} [ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources] Critical system error detected. Will be handled accordingly to configured handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext [type=CRITICAL_ERROR, err=java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be cast to class o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap'; o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in unnamed module of loader 'app')]] java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be cast to class org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap'; org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in unnamed module of loader 'app') at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221) ~[classes/:?] at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) ~[classes/:?] at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) ~[classes/:?] {code} Even we fix this, we fail further at: {code:java} private GridKernalContext standaloneKernalContext(File dumpDir, IgniteLogger log) { File binaryMeta = CacheObjectBinaryProcessorImpl.binaryWorkDir(dumpDir.getAbsolutePath(), F.first(metadata).folderName()); File marshaller = new File(dumpDir, DFLT_MARSHALLER_PATH); A.ensure(binaryMeta.exists(), "binary metadata directory not exists"); {code} If cache with node filter was created by a client or not by the excluded node (or may be not from the coordinator), it is empty. Stacktrace: {code:java} Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is invalid: binary metadata directory not exists at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617) at org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325) at org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254) at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) at org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) ... 3 more Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid: binary metadata directory not exists at org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169) at org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.<init>(Dump.java:149) at org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370) ... 15 more {code} > Cache dump fails on a cache with a node filter. > ----------------------------------------------- > > Key: IGNITE-21656 > URL: https://issues.apache.org/jira/browse/IGNITE-21656 > Project: Ignite > Issue Type: Bug > Reporter: Vladimir Steshin > Priority: Minor > Time Spent: 10m > Remaining Estimate: 0h > > Ignite may fail to create dump of cache with a node filter. > 1) If there are no partitions on some node, we assign the empty set as the > snapshot task result: > {code:java} > if (parts.isEmpty() && !withMetaStorage) > task0 = new GridFinishedFuture<>(Collections.emptySet()); > else { > ... > } > {code} > Cache dump doesn't contain the metastorage. Further, we may cast the result to > {code:java} > SnapshotFutureTaskResult res = (SnapshotFutureTaskResult)task0.result(); > {code} > but it is a > {code:java} > Collections#EMPTY_SET > {code} > Stacktrace: > {code:java} > [ERROR][snapshot-runner-#413%dump.IgniteCacheDumpSelfTest0%][IgniteTestResources] > Critical system error detected. Will be handled accordingly to configured > handler [hnd=NoOpFailureHandler [super=AbstractFailureHandler > [ignoredFailureTypes=UnmodifiableSet [SYSTEM_WORKER_BLOCKED, > SYSTEM_CRITICAL_OPERATION_TIMEOUT]]], failureCtx=FailureContext > [type=CRITICAL_ERROR, err=java.lang.ClassCastException: class > java.util.Collections$EmptySet cannot be cast to class > o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult > (java.util.Collections$EmptySet is in module java.base of loader 'bootstrap'; > o.a.i.i.processors.cache.persistence.snapshot.SnapshotFutureTaskResult is in > unnamed module of loader 'app')]] > java.lang.ClassCastException: class java.util.Collections$EmptySet cannot be > cast to class > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult > (java.util.Collections$EmptySet is in module java.base of loader > 'bootstrap'; > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotFutureTaskResult > is in unnamed module of loader 'app') > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1221) > ~[classes/:?] > at > org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) > ~[classes/:?] > at > org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) > ~[classes/:?] > {code} > 2) Even we fix the cast, further failure appears: > {code:java} > private GridKernalContext standaloneKernalContext(File dumpDir, IgniteLogger > log) { > File binaryMeta = > CacheObjectBinaryProcessorImpl.binaryWorkDir(dumpDir.getAbsolutePath(), > F.first(metadata).folderName()); > File marshaller = new File(dumpDir, DFLT_MARSHALLER_PATH); > A.ensure(binaryMeta.exists(), "binary metadata directory not exists"); > {code} > If cache with node filter was created by a client or not by the excluded node > (or may be not from the coordinator), it may be checked before any dump data > are written. > Stacktrace: > {code:java} > Caused by: class org.apache.ignite.IgniteException: Ouch! Argument is > invalid: binary metadata directory not exists > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:382) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.invoke(SnapshotPartitionsVerifyHandler.java:183) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:62) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsQuickVerifyHandler.invoke(SnapshotPartitionsQuickVerifyHandler.java:37) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invoke(IgniteSnapshotManager.java:3396) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.lambda$invokeAll$8d56ceed$1(IgniteSnapshotManager.java:3328) > at > org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11715) > at > org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11617) > at > org.apache.ignite.internal.util.IgniteUtils.doInParallel(IgniteUtils.java:11597) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager$SnapshotHandlers.invokeAll(IgniteSnapshotManager.java:3325) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.IgniteSnapshotManager.lambda$initLocalFullSnapshot$923db49$1(IgniteSnapshotManager.java:1254) > at > org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$d8c8d23e$1(GridFinishedFuture.java:180) > at > org.apache.ignite.internal.util.future.GridFinishedFuture.lambda$chain$0(GridFinishedFuture.java:163) > ... 3 more > Caused by: java.lang.IllegalArgumentException: Ouch! Argument is invalid: > binary metadata directory not exists > at > org.apache.ignite.internal.util.GridArgumentCheck.ensure(GridArgumentCheck.java:110) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.standaloneKernalContext(Dump.java:169) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.(Dump.java:149) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.SnapshotPartitionsVerifyHandler.checkDumpFiles(SnapshotPartitionsVerifyHandler.java:370) > {code} > 3) We might fix this with creating normal dump task instead of > {code:java} > task0 = new GridFinishedFuture<>(new > SnapshotFutureTaskResult(Collections.emptySet(), null)) > {code} > It could write the nessesary dump content before checking it. > But one more fauilure is met: > {code:java} > Caused by: class org.apache.ignite.IgniteException: Wrong number of group > directories: 0 > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.dumpGroupDirectory(Dump.java:384) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.Dump.configs(Dump.java:237) > at org.apache.ignite.dump.DumpReader.lambda$run$2(DumpReader.java:107) > at > java.base/java.util.stream.ReferencePipeline$7$1.accept(ReferencePipeline.java:271) > at java.base/java.util.HashMap$EntrySpliterator.tryAdvance(HashMap.java:1785) > at > java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.lambda$initPartialTraversalState$0(StreamSpliterators.java:294) > at > java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.fillBuffer(StreamSpliterators.java:206) > at > java.base/java.util.stream.StreamSpliterators$AbstractWrappingSpliterator.doAdvance(StreamSpliterators.java:161) > at > java.base/java.util.stream.StreamSpliterators$WrappingSpliterator.tryAdvance(StreamSpliterators.java:300) > at java.base/java.util.Spliterators$1Adapter.hasNext(Spliterators.java:681) > at java.base/java.util.Iterator.forEachRemaining(Iterator.java:132) > at > org.apache.ignite.internal.processors.cache.persistence.snapshot.dump.AbstractCacheDumpTest$1.onCacheConfigs(AbstractCacheDumpTest.java:370) > at org.apache.ignite.dump.DumpReader.run(DumpReader.java:106) > ... 12 more > {code} > There is no cache group directory at all on the filtered node. -- This message was sent by Atlassian Jira (v8.20.10#820010)